Leveraging Existing Tests in Automated Test Generation for Web Applications
•
1 like•1,794 views
This document describes a technique called Testilizer that combines manual and automated testing of web applications. Testilizer leverages existing manual test cases to generate new test cases by exploring the extended state flow graph in a way that stays close to the original manual test paths. It also regenerates assertions for the new test cases by reusing, exactly matching, or generating similar assertions to those in the original test cases. An evaluation of Testilizer showed it was able to generate test cases that improved code coverage while making significant use of input data, sequences, and assertions from the original manual test cases.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Leveraging Existing Tests in Automated Test Generation for Web Applications
Similar to Leveraging Existing Tests in Automated Test Generation for Web Applications (20)
More from SALT Lab @ UBC
More from SALT Lab @ UBC (9)
Recently uploaded
Recently uploaded (20)
Leveraging Existing Tests in Automated Test Generation for Web Applications
- 1. Leveraging Existing Tests in Automated Test Generation for Web Applications Amin Milani Fard Mehdi Mirzaaghaei Ali Mesbah ASE 2014
- 2. 2 Testing web applications
- 3. 3
- 4. Manual DOM-based testing Very useful because of the domain knowledge of test developers, but … 4 Incomplete
- 5. Automated testing Challenges of automated methods: • Proper input values • Choosing paths to explore • Generating effective assertions 5
- 6. Combining manual and automated tests - Input data and sequence - DOM elements to be asserted 6 - Automated crawling - Automated test case generation
- 7. 7 IDEA Initial State-Flow Graph Extended State-Flow Graph Human-written test cases Generated test cases
- 8. Testilizer: our technique and tool Eliminate failing assertions 10
- 9. Exploring alternative paths 13 • Remain close to the manual-test paths. • Randomly select a manual-test state with unexercised candidate clickables and navigates the application through that state. • Rationale: Diverging too much from the manual-test paths makes the human-written assertions too disparate and less useful.
- 10. 14 Initial State-Flow Graph Extended State-Flow Graph
- 11. Regenerating assertions Leveraging asserted DOM patterns in existing test suites: 1) Reusing the same assertion 2) Regenerating assertions for exact DOM element/region match 3) Generating assertions for similar DOM region match 15
- 12. Assertion reuse An assertion on a shared state can be reused for a new test case. 16
- 13. Assertion regeneration Repetition-based assertion regeneration 1. exact element-based assertions 2. exact region-based assertions 17 Checked Element Checked Element Region
- 14. Generating similar region assertions Inexact element/region repetition of a checked element/region can also be important for testing. 18 <div id="header"> <div id="nav"> <div id="article"> <div id="sidebar"> <div id="section"> <div id="footer"> <div id="header"> <div id="nav"> <span id="main"> <span <div id="footer"> id="menue"> <span id="content">
- 15. Learning assertions for similar regions A classification problem: • Is a block level DOM element important to be checked by an assertion? We apply machine learning to train a classier • Features: BlockCenterX*, BlockCenterY*, BlockWidth*, BlockHeight*, TextImportance, InnerHtmlLength,* LinkNum, ChildrenNum 19 * Song et al. [SIGKDD Explorations’04]
- 16. Assertion minimization Assertions should be reduced for maintainability • Avoiding redundant assertions • Prioritizing assertions 1. Original assertions 2. Reused, and exact element/region assertions 3. Similar region assertions 20
- 17. Implementation 21 Instrument Test suite State-space exploration Learning assertions
- 18. 22 Evaluation • Effectiveness of the generated assertions • Original test suite information (input data, sequences, and assertions) usage • Code coverage improvement
- 19. 23 Controlled experiment Test Suite Sequence Generation Assertion Generation ORIG Manual Manual Testilizer (EXND+AR) Traversing extended SFG EXND+RND Traversing extended SFG RAND+RND Traversing random SFG Objects: 4 open source web apps (EXND) Assertion regeneration (AR) (EXND) Random (RND) (RAND) Random (RND) Applied DOM, jQuery, and XHR mutation operators at the JavaScript code level Constraints: 5 minutes exploration time, max 5 generated assertions per state
- 20. 24 Fault detection capability
- 21. Test model and info usage stats 25 65 54 43 32 21 10 ORIG RAND EXND (ours) # States # Transitions # Test Cases 125% 100% 75% 50% 25% 0% ORIG RAND EXND (ours) Orig State Coverage Orig Transition Coverage Orig Input Data Usage Orig Assertion Usage 11% 12% 31% 45% Original Reused Exact Generated Similar Generated Avg % of generated assertions by Testilizer
- 22. Code coverage Increasing client-side code coverage was not the main goal. 26 33% 26% 20% 13% 7% 0% ORIG RAND EXND (ours) Code coverge
- 23. 27 Summary
Editor's Notes
- This is a join work with Mehdi Mirzaaghaei and Ali Mesbah at University of british Columbia, on leveraging existing test cases to generate new tests for web applications, which is also an industry demand.
- Modern web apps are composed of multiple languages such as HTML, JS, CSS, and server-side code which get rendered by a web browser to show the application’s runtime DOM. To avoid dealing with these interactions separately, developers test the correct behaviour of a web app through its manifested DOM, using testing frameworks such as Selenium or CasperJS. These frameworks provide APIs to write test scripts for functional testing of web applications.
- This is a test case for testing add note in the Organizer app.
- Manually written tests are valuable for their domain knowledge. Writing test cases manually, even by using record and replay tools, is very time consuming and the result is not complete. Therefore there is a need for automated tools.
- Many automated testing techniques are based on crawling to explore the state space of the application. Limitations: Input values: Generating proper input values (such as form data) automatically is challenging. Paths to explore: Which paths are more important for testing? Blind paths exploration may results in obtaining a partial coverage of a specific region of the application. Assertions: Generating proper assertions automatically without human knowledge is known to be challenging. Many web testing techniques rely on generic invariants or standard validators.
- Using the human domain knowledge in tests: provide valid input data know what DOM elements should be asserted and how. We propose to 1) mine the human knowledge existing in manually-written test cases 2) combine that knowledge with the power of automated crawling 3) extend the test suite for uncovered/unchecked portions of the app
- We instrument the test suite to collect information about DOM interactions such as elements accessed in actions (e.g., clicks) and assertions as well as the structure of the DOM states covered. We instrument all DOM related method calls and calls with arguments that have DOM element locaters. A test operation is a triple <action, target, input>, where action species an event-based action (e.g., a click), or an assertion (e.g., verifying a text), target pertains to the DOM element to perform the action on, and input species input values (e.g., data for filling a form). We make sure that the generated assertions are stable, i.e., do not falsely fail, when running the test suite on an unmodified version of the app. Some of these assertions are not only DOM related but also depend on the specific path through which the DOM state is reached. Our technique automatically identifies and filters these false positive cases from the generated test suite.
- We instrument the test suite to collect information about DOM interactions such as elements accessed in actions (e.g., clicks) and assertions as well as the structure of the DOM states covered. We instrument all DOM related method calls and calls with arguments that have DOM element locaters. A test operation is a triple <action, target, input>, where action species an event-based action (e.g., a click), or an assertion (e.g., verifying a text), target pertains to the DOM element to perform the action on, and input species input values (e.g., data for filling a form). We make sure that the generated assertions are stable, i.e., do not falsely fail, when running the test suite on an unmodified version of the app. Some of these assertions are not only DOM related but also depend on the specific path through which the DOM state is reached. Our technique automatically identifies and filters these false positive cases from the generated test suite.
- Regenerating assertions is a major contribution of this work. These assertions basically checks the existence of a DOM element or its attribute and textual values. Based on the concept of matching, partial matching, and similar matching, We generate new assertions by leveraging DOM patterns in test suites.
- s1 is shared among many other paths from Index to s5. Assertions on s1 can be exactly reused for generated test cases exercising alternative paths.
- Full exact match vs partial exact match region-based assertions
- Avoiding redundant assertions: Assertions that are subsumed by other assertions are redundant and safely eliminated Prioritizing assertions: We prioritize assertions in each state in the order of specification/constraint: the original assertions the reused, the exact element/region assertion assertions similar region assertions
- Our approach is implemented as an open source project available for download from out guithub repository
- To evaluate our approach we answer three research questions.
- Compared test suite generated by 1. human tester (ORIG) 2. Testilizer: traversing the extended SFG and generating assertions (EXND+AR) 3. traversing the extended SFG and random assertion generation (EXND+RND) 4. random exploration and random assertion generation (RAND+RND)
- Testilizer, on average: 12% original assertions, 11% reused assertions, 31% exact generated assertions, and 45% of similar learned assertions.
- summary