programs testing programs

programs testing programs GUI-based test automation1st Friday – 02.03.2012Sascha Bartels 1

1. What‘s „test automation“?2. Why do you do that?3. How do I know, that I‘m doing it right?4. Is our test automation cool? 2

01 | WHAT‘S „TEST AUTOMATION“? 3

A user browses with his browser on PokerStrategy.com 4

regression tests before each release of PokerStrategy.com 5

02 | WHY DO YOU DO THAT? 6

Possible objectives of test automation:- find more bugs- nightly regression tests- reduce testing staff- shorter test periods- test moreFrequent problems of test automation: [1]1. unrealistic expectations (of the management)2. poor testing processes („Automating chaos just gives faster chaos“) ... etc.Let us examine our goals more exactly!1: Software Test Automation Effective use of test execution tools – Dorothy Graham & Mark Fewster 7

find more bugsthe idea:manual testing finds bugsautomated tests are faster than manualtestsso automated tests find more bugs 8

find more bugsthe practice:- automated tests are repeated tests- looks for unexpected side effects- so automated tests can findmore regression bugs- testers have more time for manualtests- poor tests are not getting better byautomating them 9

nightly regression teststhe idea:There are unused ressources(like test systems e.g.)We could run automated tests „whilewe sleep“ 10

nightly regression teststhe practice:the correct tests have to be selectedtest results need to be analyzed 11

reduce testing staffthe idea:The automation tool costs money, so weshould be able to save somewhereelseWe want to reduce costs and staffis expensive 12

reduce testing staffthe practice:tests have to be automated and maintainedtest results must be analysedtest automation can make work easierfor testers 13

shorter test periodsthe idea:reduce deadline pressuretesting is a „bottleneck“, so we will savetime there 14

shorter test periodsthe practice:the goal can be achieved more easily:run fewer tests, omit long tests,cut regression testingAutomated tests take time: creation and maintenance of test cases analysis of test resultssoftware quality has the greatest influence on the time neededfor testing 15

test morethe idea:achieve a better test coveragetesting is good, so more testinghas to be better 16

test morethe practice:unimportant tests produce moreMaintenance instead of being gainfulsome tests require higherautomation effort,e.g. for technical reasons 17

Possible realistic objectives - find more regression bugs - automate the most importants tests - a useful amount of tests run at night and / or weekend - relieve testers and support manual testing It is important to know why you automate, so you know what and how to automate. Objectives of testing should not get mixed up with the objectives of automation. Test automation is not a tool against bad test practicesThat‘s No Reason To Automate! Why Good Objectvies Are Critical to Test Execution Automation – Dorothy Graham and Mark Fewster 18

04 | HOW DO I KNOW, THAT I‘M DOINGIT RIGHT? 19

measuringmetricsDepending on the objective and the initial situation other metrics areimportant. Does this one consume as much petrol as this? Will this car fit into my garage? Yes? There might be a hole in the tank. 20

1. DDP– Defect Detection PercentageIs the relation between found and currently known bugs.Example: A release contains 100 new bugs, the test run finds 70 bugs: DDP: 70%Weaknesses in this approach:1. The absolute amount of bugs is unknown.2. It is unknown at what time a bug has come into the product.3. Not all bugs are equally critical.4. Not all bugs are equally difficult to find.5. No reflection of the absolute count.It is easy to determine this measure and it can have a high significance. 21

1. ROI – Return on investment for process optimization current process optimized processCosts of test execution 10 000 € 20 000€DDP 70% 90%found bugs 700 900bugfixing (100 € per bug) 70 000 € 90 000 €bugs found after release 300 100bugfixing after release (1000 € per bug) 300 000 € 100 000 €total costs 380 000 € 210 000 €savings: 170 000 € investment: 10 000€ROI (savings/investment): 1700% 22

1. ROI – Return on investment in test automation manual testing automated testingCosts for test case design 6 000 € 6 000 €costs for automation 0 16 000 €total one-time expenses 6 000 € 22 000€costs for test execution 5 000 € 1 000 €amount of test execution cycles 3 3costs per release 15 000 € 3 000 €total costs of 4 releases a year 66 000 € 34 000 €savings per year: 32 000 € investment: 16 000€ROI (savings/investment): 200% 23

My test automationis much cooler than yours! 24

How do you get that idea?- I have a lot of test cases- which run every night for several hours- and provide lots of test results 25

- Well, I have:- saved more hours of manual testing(effectiveness)- spent less time on test casemaintenance (maintainability)- more bugtickets per fault report(reliability)- less training time for new employees(usability) 26

Hm…- I need to edit many tests for eachrelease, so that they can beexecuted- I need 4 hours a day to analysemy logs files- if I find a bug, it is most likelycaused by my test environment- my new colleagues need 2 weeksof training before they can help me- and after every release we haveto fix many bugs that were missedduring testing 27

And then I have:- less effort to find the test case thattests a specific range (flexibility)- fewer test cases blocked by the samesoftware bug (robustness)- less effort to run test cases ondifferent test environments(portability) 28

05 | IS OUR TEST AUTOMATION COOL? 30

We have 18 test suites for:- conversion path tests, submit poker accounts, support ticket system, fraud check, upload images, buy points packages, video section…- 865 performed tests- 9 hours test run every nightHowever, we have two test systems (integration and system test) and almostall tests run in Internet Explorer and Firefox.- about 3400 test executions- 36 hours of test runs in total- 5 virtual machines for test execution 31

1. maintainabilitycentral master suite with:- object repository- procedures for common test steps (e.g. login) 32

2. efficiency- before the introduction of test automation: 12 hours of manual regression testing for each release and test environment- today: 4 hours of manual regression testing for each release and test environment3. reliabilitysolved problems with external dependencies (Facebook, Google ...)- requests are routed via the host file on an internal web server- answers with 1x1 pixel images or empty responses to scripts 33

4. flexibility- a test suite per functional section- each test suite can be started by pressing a button in Jenkins- free choice of browser and the test environment5. usability- graphical abstraction of test scripts in QF-Test- using Jython or Groovy scripts (database access, HTML)6. robustness- 3 test suites are currently on “failed", there are 2 bugtickets- a test suite is unfortunately affected by a bug in Firefox 34

7. portability- choice of the test environment and the browser via parameter- at the moment QF-Test supports only Internet Explorer and Firefox 35

Any questions? 36

ICANS GmbHValentinskamp 1820354 HamburgGermanyPhone: +49 40 22 63 82 9-0Fax: +49 40 38 67 15 92Web: www.icans-gmbh.com 37

http://devblog.icans-gmbh.com	3293
http://www.icans-gmbh.de	447
http://icans-gmbh.de	141
http://blog.icans-gmbh.com	11
http://translate.googleusercontent.com	3
http://ranksit.com	1
http://html2canvas.hertzen.com	1
http://www.techgig.com	1

programs testing programs

by ICANS GmbH

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 3,898

Statistics