![Why do we need automated UI test case repair?
Web
application
Test
cases
Write
Tester
s
Click
2
Click
By.xpath("//a[text()='Admin']")](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
Similar to web test repair.pptx
Similar to web test repair.pptx (20)
Recently uploaded
Recently uploaded (20)
web test repair.pptx
- 1. Automated Fixing of Web UI Tests via Iterative Element Matching 1 Yuanzhang Lin∗ Beihang University Beijing, China linyz2020@gmail.com Guoyao Wen Huawei Technologies Co., Ltd. Shenzhen, China wenguoyao@huawei.com Xiang Gao Beihang University Beijing, China xiang_gao@buaa.edu.cn ∗This work was primarily undertaken by the author during his internship at Huawei while studying at the Southern University of Science and Technology. We greatly appreciate the suggestions and effort provided by Shin Hwei Tan. ASE 2023
- 2. Why do we need automated UI test case repair? Web application Test cases Write Tester s Click 2 Click By.xpath("//a[text()='Admin']")
- 3. Why do we need automated UI test case repair? Web application Test cases Write Tester s Click Find the new locator. Modify the broken tests. Make sure the repaired test cases run correctly. 3 Click By.xpath("//a[text()='Admin']") By.xpath("//a[text()=‘Rooms'] ")
- 4. Why do we need automated UI test case repair? Web application Test cases Write Tester s Click Manual test case repair is time-consuming and labor-intensive. There is a hope for automated tools to assist in repairing test cases. Find the new locator. Modify the broken tests. Make sure the repaired test cases run correctly. 4 Click By.xpath("//a[text()='Admin']") By.xpath("//a[text()=‘Rooms'] ")
- 5. Why do existing algorithms have insufficient matching accuracy? driver.findElement(By.xpath("/html/body/form/input[3]")).click(); Thread.sleep(2000); driver.findElement(By.xpath("//a[text()='Admin']")).click(); .... Assert.assertEquals(driver.findElement(By.xpath("//a[@href='admi n.php']")).getText(), "Back to Admin"); 2 1 3 1 version 1.2.6.1 Part of the broken test case “DeleteNegativeAreaTest”. 5 3
- 6. Why do existing algorithms have insufficient matching accuracy? driver.findElement(By.xpath("/html/body/form/input[3]")).click(); Thread.sleep(2000); driver.findElement(By.xpath("//a[text()='Admin']")).click(); .... Assert.assertEquals(driver.findElement(By.xpath("//a[@href='admi n.php']")).getText(), "Back to Admin"); 2 1 3 1 1 version 1.2.6.1 version 1.4.9 1. Visual change (background color). 2. Text change. 3. Properties change. Part of the broken test case “DeleteNegativeAreaTest”. Self-information: 6 3 3 Part of HTML in the version 1.2.6.1 Part of HTML in the version 1.4.9
- 7. Why do existing tools have insufficient matching accuracy? To improve accuracy , SFTM simply uses the parent node’s properties to assist in matching the child nodes. 7 [1] Brisset, Sacha, et al. "SFTM: Fast matching of web pages using Similarity-based Flexible Tree Matching. " Information Systems 112 (2023): 102126. Part of HTML in the version 1.2.6.1 Part of HTML in the version 1.4.9 the properties of parent nodes have also changed
- 8. How would a human match these two elements? 1 Match elements with high similarity: 8 Matching elements by multiple steps.
- 9. How would a human match these two elements? 1 2 Match elements with high similarity: Match the group of three elements because two elements in this group have already been matched: 9 Matching elements by multiple steps.
- 10. How would a human match these two elements? 1 2 3 “admin.php?day=28;month=01” Match elements with high similarity: Match the group of three elements because two elements in this group have already been matched: Infer that Admin and Rooms button match because they have certain similar properties and are located within the matching region: 10 Matching elements by multiple steps.
- 11. How would a human match these two elements? I1: Elements with closer matched ancestors are more similar. I2: Ancestors with more common leaf elements are more similar. I3: Matching elements by multiple iterations. Our tool UITESTFIX based on the following idea: 11 ancestor leaf elements close r 1 1 2 3 2 We further propose two similarity based on I1 and I2, which will updated iteratively.
- 12. Id Similarity. The most straightforward similarity metric is the exact matching of n.id. To calculate the similarity of text properties, we use Levenshtein distance. Property Similarity. We use TF-IDF to calculate property similarity, where text, tag and attribute are used as tokens. Text Similarity. 01 02 03 Initialize the similarity matrix. 12 Help Help Levenshtein distance Old version New version TF-IDF id="chips" id="chips" Exact matching Methodology - Similarity Metrics
- 13. 13 Closest matched ancestors The length from the root node element to the closer matched ancestors. Closer matched ancestors Matching relationship I1: Elements with closer matched ancestors are more similar. Methodology - Path Similarity
- 14. Region Similarity： The proportion of matching leaf node elements in two region elements Old Region Element： New Region Element： 14 matched Old leaf node elements: New leaf node elements: More matched leaf elements I2: Ancestors with more common leaf elements are more similar. Methodology - Region Similarity
- 15. 15 Other elements… Other elements… I3: Matching elements by multiple iterations. Methodology - Iterative matching process.
- 16. 16 1 1 1 1 1 Other elements… Other elements… I3: Matching elements by multiple iterations. Methodology - Iterative matching process.
- 17. 17 1 1 1 1 1 Other elements… Other elements… I3: Matching elements by multiple iterations. Methodology - Iterative matching process.
- 18. 18 1 1 1 1 1 2 2 2 2 Other elements… Other elements… I3: Matching elements by multiple iterations. Methodology - Iterative matching process.
- 19. 19 1 1 1 1 1 2 2 2 2 3 3 Other elements… Other elements… The iteration stops at the fourth round since the matching results are no longer updated. I3: Matching elements by multiple iterations. Methodology - Iterative matching process.
- 20. Grouping old new Path Similarity Region Similarity Match Result Dynamic Repair GetTopMatch Id Similarity Prop Simialrity Text Similarity Element Match Test Query Iterative NoSuchElementException InvalidElementStateException Execute the broken action on the matched element to complete the dynamic repair. Build the initial similarity matrix. Iteratively updates similarity. Outputs the final matching results. 20 Methodology - How does UITESTFIX work?
- 21. Evaluation UI Match Dataset: UI Test Dataset: Effectiveness on Matching Elements. Effectiveness on Repairing UI Tests. [1] F. Shao, R. Xu, W. Haque, J. Xu, Y. Zhang, W. Yang, Y. Ye, and X. Xiao, “Webevo: taming web application evolution via detecting semantic structure changes,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021, pp. 16–28. 21
- 22. Evaluation Effectiveness on Matching Elements. 22 Publicly available web pages. Industrial web pages.
- 23. Evaluation UITESTFIX improved by 16.1% compared to the best existing algorithm. With improved accuracy, we believe it is acceptable to take <1s to generate matching results for each web page. Effectiveness on Matching Elements. 23 Publicly available web pages. Industrial web pages.
- 24. Evaluation Effectiveness on Repairing UI Tests. 24 Publicly available test cases. Industrial test cases.
- 25. Evaluation In total, UITESTFIX successfully fixes 113(68%) broken tests, while the best existing tool only fixes 73(44%) of them. Effectiveness on Repairing UI Tests. 25 Publicly available test cases. Industrial test cases.
- 26. Conclusion 02 A novel iterative matching algorithm. We evaluate algorithms on publicly available and industrial datasets 26 01
Editor's Notes
- Hello! everyone. I'm yuanzhang lin from beihang university. I'm going to present our work "Automated Fixing of Web UI Tests via Iterative Element Matching". This work is a collaboration between Beihang University , Sustech and Huawei. I greatly appreciate the suggestions and effort provided by my master supervsor Shin Hwei Tan.
- Generally, testers write UI test cases for testing a web application. The test case will use the locator to find the element and do action on it.
- When the version is updated, some actions will crash because the original locator cannot find the element. The testers needs to find the new locator, use it to replace the crashed locator, and make sure the repaired test cases run correctly.
- But manual test case repair is time-consuming and labor-intensive. These is a hope for automated tools to assist in repairing test cases.
- There are currently some existing work for UI test cases, and we hope to use an example to explain why existing algorithms have insufficient matching accuracy. This is a part of the broken test case. The test case will click the Admin button in the third step.
- However, in the new version, the "Admin" button has changed to the "Rooms" button. Some information of the element, such as visual information, text, and properties, has been changed. For example, the background color of this button changes from light blue to dark blue. Therefore, some exsiting algorithms , such as WATER、 VISTA and WEBEVO, based on these information can not match this element.
- To improve accuracy , SFTM simply uses the parent node's attributes to assist in matching the child nodes. In this picture, parent node is the td element， Since the td element in the new version has no properties, the similarity between td elements is very low. So, the parent nodes can not help the these elements to match and SFTM fails to match these elements.
- To further improve accuracy, let us first to see how human match these two elements. Firstly, match elements with high similarity, such as the Help element and the Report element.
- Secondly, Match the group of three elements because two elements in this group have already been matched.
- Finally, infer that these two buttons match because they have certain similar properties and are located in the matching regions. We can find that human match element by multiple steps.
- So, Inspired by human matching methods, we proposed our tool UITESTFIX based on the following idea: The first idea is that elements with closer matched ancestors are more similar. The second idea is that ancestors with more common leaf elements are more similar. The third idea is that matching elements by multiple iterations. We further propose two similarity based on I1 and I2, and them will updated iteratively.
- Next we will introduce our method UITESTFIX in more detail. At first, to evaluate similarity of two elements , we use Id similarity, Properties Similarity and Text similarity to initialize the similarity matrix.
- Beyond these three similarities, we propose two similarities to use matched elements to help the unmatched element. We proposed path similarity based on idea 1. Given the element, we first collect the path from the root node to the target element, and evaluate the similarity of two path. If the elements with closer matched ancestors, them will have higher path similarity.
- We proposed region similarity based on idea 2. Region similarity is the proportion of matching leaf node elements in two region elements. The more leaf elements matched in these two region, the higher the similarity of the region elements. For example, in this picture, the search element has been matched and it will increase the similarity of these two region elements.
- The core idea of UITESTFIX is the iterative matching process, which is based on the idea 3.
- During the first iteration, some highly similar elements successfully matched.
- Based on the existing matching results, the similarity of some elements improved. For example, the similarity of this element improved because all of its leaf nodes successfully matched.
- Therefore, these is the the matching results after the second iteration .
- The second iteration's matching results increased the path similarity of the Admin button and the input element, leading to a successful match. The iteration stops at the fourth round since the matching results are no longer updated.
- So, let's look at how UITESTFIX works in totally. When a test case throws an exception during the execution, UITESTFIX will catch this exception. Subsequently, UITESTFIX uses three similarities to build the initial similarity, iteratively updates similarity and outputs the final matching results. UITESTFIX will execute the broken action on the matched element to complete the dynamic repair.
- We use two datasets to evaluate algorithms. The UI match dataset is used to evaluate the performance of element matching, while the UI test dataset is used to evaluate the performance of test case repair.
- First, we want to know the effectiveness on matching elements. From the experimental results, we can see that UITESTFIX improved by 16.1% compared to the best existing algorithm. With improved accuracy, we believe it is acceptable to take less than one second to generate matching result.
- First, we want to know the effectiveness on matching elements. From the experimental results, we can see that UITESTFIX improved by 16.1% compared to the best existing algorithm. With improved accuracy, we believe it is acceptable to take less than one second to generate matching result.
- The table shows the effectiveness on repairing UI tests, UITESTFIX successfully fixes 113 broken test cases, while the best existing tool only fixes 73 of them.
- The table shows the effectiveness on repairing UI tests, UITESTFIX successfully fixes 113 broken test cases, while the best existing tool only fixes 73 of them.
- In conclusion, we proposed UITESTFIX, an approach based on a novel iterative matching algorithm for fixing broken UI tests. Then, we evaluate algorithms on publicly available and industrial datasets. UITESTFIX outperforms four existing approaches in producing more accurate matching and correct repairs. That’s all of my presentation, thank you for your listening.