Help to release higher quality product: because more exactly: automation is more exactly than manual way. faster: test coverage is higher than human do On time?? computer execute quickly and inconsistently Sure … but you need have good plan & execution. Because when you start implementing your script, it means that you can face to solve many following automation problems goal of automation: + reduce cost when your AUT has new upgrade/version: just need to re-execute the script again to test + exact result- Problem which we try to solve in automation is how to implement your script become stable, reusable & effective
Better user of qualified in-house testing resourcesTedious happens at repetitive task and use most our resource so automation release tester for creative tasksCan’t test manuallyLoad test, performance test, stability testAutomation save time, money and release manual testing skillExplain reason- Load test, performance test,
Smoke, user acceptance, functional test need to retestAutomate testing only do paths I indicate but tester can work around the paths to discover more bugsHow many bugs might I lose and when
Follow the idea of slideThe talent tester do automation or maintenance scripts then there isn’t enough good person do the test => test productivity decreasesOn the side la song song hay congviecphu
Follow the idea of slideNeed explain moreWhat is good-enough automationTightening the testing cycle
Give explanation of this comparison
Run Automatic totally1. Run unit test; 2. Build new version; 3. Check new build; 4. Download; 5. Remove old build, install new build6. Run smoke test and email result to everyone related; 7. Run another test typeReusable and scalable if architected effectivelyArchitected effectively: Module, Action , Interface, …Scalable: when logic change we just modify at one position
This point can regard as the weak of automation, follow the idea of the slideUncertainty and lack of controlWhen program change much cause the application of automation difficultlyFocus on making the test run instead of quality of testWhat is quality of test?
Follow the slide, give example for this.Some reasons can become from the client – customer, they want us to test application as soon as possible, so we didn’t have time to implement script… just focus on manual test to report bugs as much as possible- usability: nice, fast, …
Follow the slideApplication should be stableis at phase system and user acceptanceTools for stress, load and performance testRobust script of IBMMS VS test team support unit, performance testBooksTesting computer software
Automation delivery summary reportTest scopeWhat completedWhat we seeTest case amountComplete percentTotal run time of automation per computerRegression on function coverRun result on each environment
Follow the slide & give example via any automation tool (such as: TA)
Explain the diagram, the way to work
Follow the slideRecord/playback bases on coordinate.You can give some questions to your class, also.
This is the generation 2
Follow the slide
Win runner application
This is the generation 3- There is no explain on this slide
Follow the slide- Not scalable: logic changed must be maintain script
Challenge of data driven (generation 3)Response slowly to rate of software changesLow reusability/scalability: logic in programming
Follow the slide ( try a demo via TA tool)
Follow the slideTest engineers are black-box testersAutomation engineers are programmers
Compare some testing tool. Can give question to learner: Have you ever used any testing tool in this list? Or have you ever heard about them?winRunner is previous one of QTPTest director is previous one of Quality Center
Give summary to your class to close this chapter.Some questions you can give: - How many testing generation? 4 - Could you please list out the name of them? Record & play back, scripting, data driven, action keyword - Which strong point / weak point of automation?…
1. CHAPTER 10 An Introduction toAutomated Software Testing
2. CHAPTER 10 OUTLINE• What is Test Automation?• The Benefits and Pitfalls of Test Automation• When to Use Test Automation• Test Automation Production Process• The Evolution of Test Automation
3. CHAPTER 10 OBJECTIVES• Present an overview of test automation engineering• Understand the pros and cons of test automation• Test automation strategies, tactics and implementation
4. An Introduction to CHAPTER 10.1 Automated Testing10.1 What is Test Automation?10.2 The Benefits and Pitfalls of Test Automation10.3 When to Use Test Automation10.4 Test Automation Production Process10.5 The Evolution of Test Automation
5. What is Test Automation? CHAPTER 10.1• Using computer time to execute of tests.• Commonly, test automation involves automating existing and sometimes, new tests that otherwise, will be executed by a human tester
6. Why Automate Testing?• People might say, “Automation will help us release a higher quality product on time.” – What does that mean?• What’s the goal of instituting an Automation program? – What problem are we trying to solve?
7. Why Automate Testing?– Not enough test time in schedule for repetitive tasks– Very repetitive tasks– Too many releases to test– Better use of qualified in-house testing resources– Reduce cost of testers– Can’t test manually– Optimize tester time because there is not enough time for mundane tests– Human tester can focus on testing that requires complex analysis– Save money– Save time
8. Why Automate Testing?– If I automate testing, what manual testing will I lose? How many bugs might I lose and when? Remember: Bugs are more expensive to fix later.
9. Why Automate Testing?– Often, if the manual test team also does automation, it is the best or most technically talented testers who script. – The early part of application testing is when the most talented testers are needed.• This is particularly true with script maintenance. Some talented person on the main product team is often in charge of maintaining the automation project. – Will we lose finding good early bugs because of this?– Testing cannot be an “on the side” job. You can’t write good scripts while you wait to get a build or have spare time!
10. Goal of AutomationSometimes, the goal of a test automation program can beGood-Enough Automation. Just like with the products we sell,quality is a combination of reliability, features and timeliness.If your goal of automation is tightening the testing cycle. ANDyou can also improve the test battery, you have done a goodjob.
11. Manual vs.CHAPTER 10.1 Automation Testing Manual Testing Automation Testing Test execution time Test execution time Regression test coverage Regression test coverage Responsive and flexible Relatively inflexible Human variation Machine consistency No maintenance needed High maintenance needed Immediate dividend paid Long-term dividend Not cost effective for high Not cost effective for low volume regression volume regression
12. An Introduction to CHAPTER 10.2 Automated Testing10.1 What is Test Automation?10.2 The Benefits and Pitfalls of Test Automation10.3 When to Use Test Automation10.4 Test Automation Production Process10.5 The Evolution of Test Automation
13. Test AutomationCHAPTER 10.2 Benefits• Run automated tests unattended (overnight)• Reliable, consistent repetitive testing• Increased speed in test execution• Improves quality, increased test coverage• Reduces costs and time for regression testing• Optimize the use of test resources to focus on bug findings• Reusable and scalable if architected effectively• Execution of tests that can’t be done manually• Performance testing to assess software stability, concurrency, scalability….
14. Test Automation CHAPTER 10.2 Pitfalls• Uncertainty and lack of control.• Poor scalability and maintainability. – Too sensitive to changes – Poor reusability• Low test automation coverage – Too much programming• Poor methods and disappointing quality of tests – Focus of tools instead production volume – Focus of making the tests run instead of quality of the tests• Technology vs. people issues – Gaps in platform technology support – Gaps in human resource equipped with test automation skills
15. An Introduction to CHAPTER 10.3 Automated Testing10.1 What is Test Automation?10.2 The Benefits and Pitfalls of Test Automation10.3 When to Use Test Automation10.4 Test Automation Production Process10.5 The Evolution of Test Automation
16. Which Test CasesCHAPTER 10.3 Can’t be Automated?• Usability testing "How easy is the application to use?“• One-time testing• “ASAP” testing “We need to test NOW!”• Ad hoc/random testing Based on intuition and knowledge of application• Tests without predictable results
17. The Criteria toCHAPTER 10.3 Apply Test Automation• Customer requests• Application should be stable• Tests that need to be run for every build of the application (sanity check, regression test)• Tests that use multiple data values for the same actions (data driven tests)• Tests that require detailed information from application internals (memory, CPU)• Tests that require considerable amount of time to perform manually• Stress, load or performance testing
18. An Introduction to CHAPTER 10.3 Automated Testing10.1 What is Test Automation?10.2 The Benefits and Pitfalls of Test Automation10.3 When to Use Test Automation10.4 Test Automation Production Process10.5 The Evolution of Test Automation
19. Test AutomationCHAPTER 10.4 Production Process Repeat steps Wait forInput user Check AUT until all to-be- processing toactions function outputs executed tests complete are completed 1 2 3 4Manual SynchronizeGenerate script playback Run tests orautomated to application Add checks suites of testsscripts output from 1 each step 2 3 4Automated
20. An Introduction to CHAPTER 10.5 Automated Testing10.1 What is Test Automation?10.2 The Benefits and Pitfalls of Test Automation10.3 When to Use Test Automation10.4 Test Automation Production Process10.5 The Evolution of Test Automation
21. A Brief HistoryCHAPTER 10.5 of Test AutomationSoftware Test Automation has evolved through several generations,with many improvements along the way: 1st Generation: Record and Playback 2nd Generation: Scripting 3rd Generation: Data-driven 4th Generation: Keyword-driven
22. Record & PlaybackCHAPTER 10.6 The automation tool records the steps of the user executing a test, and the tester configures some checks. The automation tool generates a script based on the recorded actions. The automation tool performs a playback of the recorded test. Even small changes to the application under test require manual re-recording of the test. → This first generation is neither efficient nor scalable.
23. Record/Playback Recorded test • Actions • Checks scripts• One time recording of Playback actions and checks• Multiple time playback Test Tool Log (Invisible to the target system) Record Playback • Actions • Checks • Results Test data entered by Target testers or users System/AUT
24. EXAMPLE A Recorded Scriptselect window "Logon"enter text "username", "administrator"enter text "password", "testonly"push button "Ok"select window "Main"push button "New Customer"expect window "Customer Information"select field "Name"type "Jones"select field "First Name"type "John"select field "Address"type "54321 Space Drive" . . .
25. EXAMPLE A Recorded ScriptCHAPTER 10.6• Dependent on screen coordinates # Analog Recording move_locator_track (1); mouse movement mtype ("<T55> <kLeft>-<kLeft>+"); mouse click type ("<t3>output14" ); move_locator_track (2); mtype ("<T35><kLeft>-<kLeft>+"); output14 enter text pause before from mouse click keyboard
26. R/P Pros and Cons• Pros – A useful way to learn about the testing tool and automation• Cons – High maintenance cost dealing with changes in the target system of AUT – Poor accessibility to test cases – Greatly affected by the working system or environment – Only suitable for GUI systems
27. Scripting Approach• Automation is regarded as a programming activity• Parameterize hard-coded values• Separate data from code by moving variables to include files• Create utility functions to be shared• Produce and maintain like any other software• Train black-box test engineers to run the scripts
28. Scripting Approach• Pros – Reusability of scripts for common tasks – Improved maintainability• Cons – Poor accessibility to test cases because tests are programmed or scripted – Greatly affected by a working system or environment – Test implementation must be done by programmers – Translation of test cases—3000 tests mean 3000 automated scripts – Manageability is still an issue due lack of visibility
29. EXAMPLEFunction EnterCustomerJones Logon Press "New Customer" Enter Field "Name", "Jones" Enter Field "First Name", "John" Enter Field "Address", "54321 Space Drive" ... LogoffEnd FunctionFunction OrderProduct Logon Press "New Order" Enter Field "Product", “OurProduct 1.0" Enter Field "Amount", "35" Enter Field "Delivery", "asap" ... LogOffEnd Function...
30. Scripting – Example ScriptCHAPTER 10.6Silk Test Script:
31. (Scripting) Data-Driven Approach• Take advantage of tester’s familiarity with test case design and creation using tables and matrices• Accommodate localization projects• Recognize the importance of patterns in test cases• Enable testers to catalog test cases with spreadsheets in software such as Excel• Enable testers to specify expected results in spreadsheets
32. EXAMPLE GOODMORNING goodmorning GoodMorningInput Data
33. EXAMPLE • File with test data: • Text file used in the test:nr case whole pattern matches goodmorning 1 off off GOODMORNING 4 2 off on GOODMORNING 3 GOODMORNING 3 on off GOODMORNING 1 GoodMorning 4 on on GOODMORNING 1 GoodMorningVietnam 5 off off goodmorning 4 6 off on goodmorning 3 • Data driven script: 7 on off goodmorning 1 for each line in the file do 8 on on goodmorning 1 open the find dialog 9 off off GoodMorning 4 read a line from the file10 off on GoodMorning 3 use the values to fill dialog11 on off GoodMorning 2 press find button12 on on GoodMorning 1 check amount of matches close the dialog
34. (Scripting) Data-Driven Approach• Pros – Separation of action and data enables tests to receive combinations of test data – Test data can be stored in a separate file or spreadsheet for better maintainability – Improved reusability and maintainability• Cons – Actions containing business logic are in automated scripts – Maintainability of changes in business logics as well as interface is still poor – Poor visibility because accessibility to the tests is still poor – Not scalable (e.g., reaching 95% of automation)
35. Test Automation Challenges• Too much programming• Response to rate of changes (software)• Low reusability/scalability• Too many platforms/technologies• Too expensive if the returned value is low
36. Keyword-drivenCHAPTER 10.6 Tests are broken down into a series of actions representing the business processes of the SUT, such as ‘enter client’, ‘transfer’, and ‘check balance’. Individual actions are automated, not the tests. The tests are stored externally from the test scripts, typically in a spreadsheet. Keyword-based test automation’s elegant, structured approach simplifies the work even further, reducing the time and expense of test design, automation, and execution by allowing team members to focus on their areas of individual expertise. Non-technical testers and business analysts can develop executable test automation from keywords representing recognizable end-user actions.
37. How Does ABT (Action-keyword) Work? Test Cases are written or described through a series of Test Lines using parameterized and reusable Action keywords Action’s ParametersAction keyword Number Name DescriptionTest Line add product 12345678 Hammer For pounding nails Actions are created and communicate with target systems and software applications including the application under test (AUT) through one or more supported Interfaces to carry out intended tasks Note: For now, think of an Action as a verb or something that you do Information about Interfaces such as Windows, Web or Java graphical user interface (GUI) or command-line interface (CLM) are later defined to enable actions to send inputs to and receive outputs from the target system and/or AUT
38. Overcome Challenges w/ Action Based Testing• Separation of action, interface, logic and data• Test data can go in a separate file or spreadsheet• Test logic can go in a separate file or spreadsheet• Interface can go in a separate file or spreadsheet• Action can also go in a separate file or spreadsheet• Action can be reused
39. Automation Approach ComparisonCHAPTER 10.6 Workload Allocation Approach Test Data Automation Test Case Construction / Engineering Production Selection Record & N/A Test Engineer N/A Playback Automation Automation Scripting Engineer Engineer N/A Automation Automation Data-driven Engineer Engineer Test Engineer Automation Keyword-driven Engineer Test Engineer Test Engineer Note: Programming skill is required for Automation Engineer
40. Examples of Test CHAPTER 10.6 Automation ToolsLogiGear is the producer of TestArchitect™, a functional automated testing tooldesigned to help achieve the most efficiency out of action-keyword testautomation. Examples of other commercial tool vendors include: Automation HP IBM Others ToolFunctional WinRunner Functional Tester SilkTest QuickTestPro Rational Robot TestPartner TestCompletePerformance LoadRunner Performance Tester SilkPerformer AstraLoad Test WebLoad OpenSTAManagement Test Director Rational TestManager SilkCentral Test Quality Center ManagerMemory Purify JProbe Profiler
41. CHAPTER 10 SUMMARY• Test Automation is a method that uses of software to control the execution of tests.• Commonly, test automation involves automating a manual process currently in.• Test Automation has benefits such as…However, it still has some pitfalls such as…• A effective strategy of test automation includes several steps…• There are many popular automation tools. Depend on the AUT, we choose the suitable automation testing tool.
42. CHAPTER 10 QUIZ• What are the advantages and disadvantages of test automation?• What are the advantages and disadvantages of test automation generations?