Generate Robust E2E Test Cases for Template-based Web Apps

Towards the Generation of Robust E2E
Test Cases
in Template-based Web Applications
Porfirio Tramontana, Anna Rita Fasolino
University of Naples “Federico II”
48th Euromicro Conference Series on Software Engineering
and Advanced Applications (SEAA)
Maspalomas, Gran Canaria, Spain, September 1st, 2022.
SEAA 2022, Maspalomas, September 1st, 2022

Motivations
◼ Capture and Replay Testing of Web
applications
➢ Effective and efficient solution for E2E testing
➢ Support in test scripting generation
➢ Automation of test case execution
➢ Requires limited programming and testing
skills
 Test case fragility
SEAA 2022, Maspalomas, Gran Canaria, September 1st, 2022

Test case fragility: examples
◼ We want to register the click on the Mark as Done button
◼ Possible locator suggested by Selenium:
◼ xpath = .btn-success
 Fragile!
◼ May broke for style sheet changing
◼ May be ambiguous for buttons having the same style

◼ xpath = (//button[@type='button'])[4]
 Fragile!
◼ May broke with button position changes

◼ xpath = //button[3]
 Fragile!
◼ May broke with button position changes
◼ May broke for table redesign

◼ xpath = //button[contains(.,'✔ Mark as Done')]
 Fragile!
◼ May broke for localization/changing in text
◼ May broke for button type changing
◼ May broke for character code changing

◼ Possible locator suggested by Katalon Recorder:
◼ xpath=(.//*[normalize-space(text()) and normalize-space(.)='✘
Remove'])[1]/following::button[1]
 Fragile!
◼ May broke for localization/changing in text
◼ May broke for button type changing
◼ May broke for character code changing

Possible solutions in literature 1/3
◼ Generation of robust locators
◼ Heuristic rules are used to provide a ranking
between a set of possible locators, based on
different possible features (text, relative
position of tags, …)
◼ M. Leotta et al., 2014. Reducing web test cases
◼ M. Leotta et al., 2016. Robula+: An algorithm for generating robust XPath locators for
web testing.
 Problem: Heuristic locators are fragile with
respect to some specific maintenance
interventions

◼ Automatic repairing of broken locators
◼ By maintaining different valid locators and
dynamically repairing broken test cases by using
alternative locators
◼ M. Leotta et al. 2015. Using Multi-Locators to Increase the Robustness of Web Test Cases
◼ I. Aldalur et al., 2020. ABLA: An Algorithm for Repairing Structure-Based Locators Through
Attribute Annotations.
 Problem:
◼ These solutions need a support to maintain
alternative locators / substitute them to broken
ones

◼ Preventive improvement of Web
application testability
◼ The developer may add identifiers while
implementing web pages in order to assure
their reliable identification
◼ Bajaj et al. 2015. Synthesizing Web Element Locators
◼ Bajaj et al. 2016. LED: Tool for synthesizing web element locators
 Problems:
◼ Not applicable on existing web sites
◼ Duplication of identifiers on dynamically
generated client pages

Proposed Solution : context
◼ MVVM Web applications
◼ Angular / Smarty / Freemarker / Twig
◼ Separation of UI design and logic with HTML
templates
◼ Template source code available at development time
◼ Dynamic generation of tags on the client side
◼ E.g. with ng- attributes (Angular)
◼ Use of C&R tools supporting design of new
locator generators
◼ E.g. Katalon Recorder

Our approach
1. Automatic injection of hooks in the
source code of web templates and html
pages
2. Automatic generation of locators based
on injected hooks
◼ Integrated with C&R tools (e.g. Katalon
Recorder)
3. Continuous integration life cycle
◼ Supported by Github

<ui - view >
...
<table >
...
<tbody >
<tr ng - repeat =" task _in_ ctrl . tasks ">
...
<td >
<div class ="form - group ">
<button type =" button " ng -if="!! task . done " ng - click =" task . done _=_ false ">
&# x270E ; Mark as To Do </ button >
<button type =" button " ng -if="! task . done " ng - click =" task . done _=_true ">&#
x2714 ; Mark as Done </ button >
<button type =" button " ng -if="! task . done " ui - sref =". edit ({ taskIndex :_
$index })">&# x270E ; Edit </ button >
<button type =" button ">&# x2718 ; Remove </ button >
</ div >
</td > </tr > </ tbody > </ table >
</ui - view >
Example Source code : Angular
<html >
...
<body ng - app =" myApp ">
<div class =" container ">
<ui - view ></ui - view >
</ div >
...
</ body >
</ html >
index.html
template.html
Item to be located

Example 1/3
◼ A robust locator must identify the involved tag
1. We inject an identifier (hook) in the tag code
◼ <button type =" button " ng -if="! task . done " ng - click =" task . done _=_ true " x-
test -hook -81 >&# x2714 ; Mark as Done </ button >
2. Since the tag is defined in the context of a template,
we have to inject another identifier on the template
definition
◼ <ui - view x-test -tpl -62 >
3. A template instance may be inserted multiple times
in a page: we have to inject another identifier into
the tag including the template
◼ <ui - view x-test -hook -9 ></ui - view >

Example 2/3
4. The tag to be located is included in a table row. Angular may
generate different rows. Our item may be identified by means
of the tr tag and the index of the selected item. We have to
inject an identifier on the table row tag
◼ <tr ng - repeat =" task _in_ ctrl . tasks " x-test -hook -73 >

Example 3/3
◼ A locator that include all these information is the
following:
◼ //*[ @x-test-tpl-1]//*[ @x-test-hook-9]
//*[ @x-test-tpl-62]//*[ @x-test-hook-73][2]
//*[ @x-test-hook-81]
◼ x-test-tpl-1 identifies the body of the index.html page
◼ x-test-hook-9 identifies the tag including the template
◼ x-test-tpl-62 identifies the definition of the template (that
includes the table)
◼ x-test-hook-73[2] identifies the second row of the table including
the button
◼ x-test-hook-81 identifies the button into the table row

<ui-view x-test-tpl-62>
...
<table x-test-hook-65>
...
<tbody x-test-hook-72>
<tr ng-repeat="task in ctrl.tasks" x-test-hook-73>
...
<td x-test-hook-79>
<div class="form-group" x-test-hook-80>
<button type="button" ng-if="!!task.done" ng-click="task.done = false" xtest-
hook-82>✎ Mark as To Do</button>
<button type="button" ng-if="!task.done" ng-click="task.done = true" xtest-
hook-81>✔ Mark as Done</button>
<button type="button" ng-if="!task.done" ui-sref=".edit({taskIndex: $index
})" x-test-hook-83>✎ Edit</button>
<button type="button" x-test-hook-84>✘ Remove</button>
</div>
</td> </tr> </tbody> </table>
</ui-view>
Hook Injected source code
<html x-test-tpl-1>
...
<body ng-app="myApp" x-
test-hook-7>
<div class="container" x-test-
hook-8>
<ui-view x-test-hook-9></ui-
view>
</div>
...
</body>
</html>
index.html
template.html

Hook Injection Algorithm
◼ For each web page or template
◼ For each template definition
◼ If the template has not yet a hook
◼ Insert x-test-tpl-(number) hook attribute
◼ For each html tag
◼ If the tag has not a yet hook
◼ Insert x-test-hook-(number) hook attribute
◼ The algorithm automatically insert hooks with increasing (number)
in order to avoid the presence of items with the same hook
◼ The algorithm is idempotent and can be run incrementally

Hook-based Locator Generation

Hook-based Locator Generation
◼ The xpath locator will include hooks from:
1. Body of the html page
2. Tags including templates
3. Template definition tags
4. Tag of the element to be located
5. In addition, if any of these tags has siblings, the
index of the tag has to be included together with the
hook of the container tag

Testing Process Integration
◼ The proposed solution can be integrated
in a E2E testing process supporting
regression testing
◼ Hook injection can be automatically executed
after each project release
◼ Test case registration with C&R tools may
generate hook-based locators

Testing Process Integration

Technical specs
◼ Hook injector has been implemented in node.js and
supports template languages having grammars defined
with TextMate (Angular, Smarty, Freemarker, Twig)
◼ Hook-based locator generation is implemented as a
script included in Katalon Recorder
◼ Katalon Recorder is able to generate executable test
cases
◼ For each test case, a headless executable version can be
automatically generated
◼ Test case execution and reporting is executed in the
context of Github by exploting Github Actions

Validation Experiment
◼ RQ1 : How do the breakages in test cases
exploiting Hook-Based locators compare
with breakages in Non-Hook-Based test
cases?
◼ RQ2 : How does the use of Hook-based
locators influence the fragility of test cases
with respect to the use of Non- Hook-
based locators?

Experimental setup
◼ Three teams of students of Advanced
Software Engineering (Laurea Degree in
Computer Engineering)
◼ They realized an Angular Web application
each, by following the proposed testing
process integration and by considering
two possible locators:
◼ Default locators provided by Katalon
◼ Hook-based locators

Experiment execution
◼ After each application release, regression testing
is executed.
◼ Some test cases broken due to locators (TKO):
◼ By obsolescence
◼ when the test scenario has been eliminated
◼ E.g. the functionality has been removed and the command
button has been deleted
◼ By fragility (TKOFRA)
◼ When modifications (in page layouts) cause the brekage of a
test case that is still needed

Results

Summary results
◼ On a total set of 214 regression test cases
◼ 16 test cases fail by obsolescence
◼ 30 test cases fail by fragility of the Default
Katalon Recorder locators
◼ No test cases fail by fragility of the Hook-
based Locators

Answers
◼ RQ1: The adoption of Hook-based locators,
instead of Default Katalon ones, significantly
reduces the percentage of broken tests in all
the considered applications, with a reduction
up to 82%.
◼ RQ2: The adoption of Hook-based locators,
instead of Default Katalon ones, drastically
reduces the fragility of test cases in all the
applications, with a reduction of 100%.

Typologies of locator breakages
◼ Attributes disappearing
◼ Text Modifications
◼ Tag type changes
◼ Index value changes
◼ except for cases involving siblings tags

Conclusions
◼ We have proposed a novel technique
supporting the generation of robust
locators for E2E testing of template based
Web applications
◼ It can be implemented in continuous
integration lifecycles

Future Works
◼ The implemented software is open source
◼ The direct link is into the paper
◼ https://sites.google.com/view/porfiriotramontana/home-page
◼ It is a prototype: we are currently experimenting its
integration in a Github supported lifecycle
◼ The validation experiment provided encouraging
results about the robustness of the generated
locators
◼ Collaborations are welcome for future
experiments and for the improvement of the
Github Actions support

Thank you for your attention!

Generate Robust E2E Test Cases for Template-based Web Apps

Recommended

Recommended

More Related Content

Similar to Generate Robust E2E Test Cases for Template-based Web Apps

Similar to Generate Robust E2E Test Cases for Template-based Web Apps (20)

More from Porfirio Tramontana

More from Porfirio Tramontana (20)

Recently uploaded

Recently uploaded (20)

Generate Robust E2E Test Cases for Template-based Web Apps