Download to read offline
Uploaded byRiccardo Coppola
Automated Generation, Evolution and Maintenance: a perspective for mobile GUI testing
The document discusses the state of Android GUI testing, outlining various techniques and classifications, including layout-based, visual testing, and methods for test generation such as scripted, capture & replay, and model-based approaches. It highlights ongoing challenges such as fragmentation and test fragility, proposing solutions like automated test maintenance, gamification in testing practices, and the use of machine learning for generating test sequences. Future work aims to integrate visual tests in continuous environments and improve test prioritization through empirical assessments and coverage scenarios.
More Related Content
Similar to Automated Generation, Evolution and Maintenance: a perspective for mobile GUI testing
![[Presentation] Automated Model-Based Android GUI Testing using Multi-Level GU...](https://cdn.slidesharecdn.com/ss_thumbnails/ymbaekase2016-presentationv1-160902084442-thumbnail.jpg?width=640&height=640&fit=bounds)
Automated Generation, Evolution and Maintenance: a perspective for mobile GUI testing
- 1. Automated Generation, Evolution andMaintenance: a perspective for mobile GUI testing Riccardo Coppola, PhD Software Engineering Research Group Politecnico di Torino, Italy riccardo.coppola@polito.it
- 2. Android GUI Testing:State of The Art
- 3. Android GUI Testing:State of the art
- 4. Android GUI Testing:State of the art The very rich UX of Android apps should encourage a thorough verification & validation
- 5. Android GUI Testing:State of the art …but this typically does not happen Source: Kochhar, Pavneet Singh, et al. "Understanding the test automation culture of app developers." 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). IEEE, 2015.
- 6. Classification of GUItesting techniques According to how the elements on the GUI are located and verified: Coordinate-Based (1° generation) Layout-based (2° generation) Visual / Image recognition (3° generation)
- 7. Layout-based GUI Testing Elementsare identified through layout properties (e.g., Ids, text, content description, widget type).
- 8.
- 9. Visual GUI Testing •Visual testing is the process of validating all the visual aspects of an application’s UI on all platforms • Goes beyond functional testing tools to ensure that things like colors, fonts, buttons are displayed correctly to the final user
- 10. Visual GUI Testing •Visual testing is based on image recognition techniques • E.g., click on a button with this appearance • Screen captures are used both as: • Locators, to identify where to perform the interactions with the GUI • Oracles, to identify what has to be checked in assertions
- 11. Visual GUI Testing Examples: Sikuli,EyeAutomate
- 12. Classification of GUItesting techniques According to how test sequences are generated: Automation APIs (scripted testing) Capture & Replay Automated input generation techniques
- 13. Scripted testing Development oftest scripts using platform-specific scripting languages. Test scripts are then executed using universal test engines (e.g., Junit) or dedicated engines.
- 14. Scripted testing Examples: Espresso,UIAutomator, Appium
- 15. Capture & ReplayTesting User inputs are given once to the user interface (capture) and then codified into a repeatable script (replay). They can be implemented by stand-alone tools or used to generate sequences for component-based testing
- 16. Capture & ReplayTesting Examples: EspressoTestRecorder (to generate Espresso component-based tests)
- 17. Model-Based GUI Testing Thetest cases are based on a Model of the user interface
- 18. Android GUI Testing:Open Issues
- 19. Android GUI Testing:Open Issues • Fragmentation • Hardware fragmentation • Software fragmentation • Test Fragility • Test of dynamic applications • Continuous Integration of GUI tests
- 20.
- 21.
- 22. Test Fragility The veryhigh variability of User Interfaces and its definition can lead to GUI Testing Fragility. A GUI automated test is said to be fragile when: • It fails when the application evolves; • The failure is not due to the modification of the functionalities under test, but to changes in the interface arrangement and definition.
- 23.
- 24.
- 25. A conceptual framework Threemain components of mobile GUI testing: • Test Generation • Test Execution • Test Maintenance
- 26.
- 27.
- 28. Test Generation Layout-based testsand visual tests should coexist
- 29. Test Generation Tooling shouldexpose multiple ways of defining test cases
- 30. Test Generation Traditional manual writingof test scripts
- 31. Test Generation • Capture& Replay tools with dynamic aids for the developer in defining test cases • Implementation of gamification concepts in the practice of GUI testing
- 32. Augmented C&R Testing+ Gamification • Proof of concept based on Scout (Nass et al.) Nass, Michel, Emil Alégroth, and Robert Feldt. "On the Industrial Applicability of Augmented Testing: An Empirical Study." 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2020.
- 33. Gamification concept • Applicationof game design concepts to other practices • To ensure… • Better performance • Better engagement
- 34. Applied gamification concepts: •Scoring mechanism and leaderboard
- 35. Applied gamification concepts: •Scoring mechanism and leaderboard • Live graphical feedback Covered Uncovered High Score
- 36. Applied gamification concepts: •Scoring mechanism and leaderboard • Live graphical feedback • «Easter Eggs»
- 37. Applied gamification concepts: •Scoring mechanism and leaderboard • Live graphical feedback • «Easter Eggs» • Achievements and customizable «player profile»
- 38. Preliminary evaluation •Coverage measuredon 10 exploratory web testing sessions without and with gamification Fulcini, Tommaso; Coppola, Riccardo; Ardito, Luca. "A Metric Framework for the Gamification of Web and Mobile GUI Testing." 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2021.
- 39. Preliminary evaluation •«Was theexperience with the gamified testing tool better than the experience with the un-gamified testing tool?»
- 40. Test Generation • Definitionof application and activity models to automatically guide test case generation • Application of machine-learning based methods for the definition of test sequences
- 41. AI-based generation • Generaltest sequences written in natural language by the tester
- 42. AI-based generation • Generaltest sequences written in natural language by the tester • Automated adaptation (to Appium test cases) and launch of test sequences to the specific app based on: • APK classification (i.e., type of app) • Activity classification (i.e., type of screen shown to the user)
- 43. AI-based generation Ardito, L.,Coppola, R., Leonardi, S., Morisio, M., & Buy, U. (2020). Automated Test Selection for Android Apps Based on APK and Activity Classification. IEEE Access, 8, 187648-187670.
- 44. AI-based generation: benefits •No need of application-specific models • Low effort in writing test case sequences • Test case sequences can be written by non-developers, and without access to the application code
- 45. Test Generation • Creationof test suites of one generation (layout-based or visual) by reuse of existing test cases of the other one
- 46. Translation-based approach • Translationfrom Layout-based test cases to Visual test cases, and viceversa; • From 2° generation properties to 3° generation screen captures; • From 3° generation screen captures to 2° generation properties.
- 47. Translation-based approach Coppola, R.,Ardito, L., Torchiano, M., & Alégroth, E. (2021). Translation from layout-based to visual android test scripts: An empirical evaluation. Journal of Systems and Software, 171, 110845.
- 48. Translation-based approach: benefits •Automate the creation of 3° generation tests from systematic reuse of 2° generation tests – and vice versa • Enhance the expressive power of existing test suites • Mitigate graphic fragilities through retranslation from the other generation equivalents
- 49.
- 50.
- 51. Test Execution • Testsshould be executed in parallel on multiple devices, both real and virtual
- 52. Parallel execution • Performance,screen rendering, properties of the visualized widgets have to be tested on all configurations
- 53. Parallel execution • IaaSsolutions can be used to provide VMs for testing Ali, A., Maghawry, H. A., & Badr, N. (2018). Automated parallel gui testing as a service for mobile applications. Journal of Software: Evolution and Process, 30(10), e1963.
- 54. Parallel execution • Existingcommercial tools (e.g., test.io) allow to crowd- source the testing, and to use real devices
- 55. Test Execution • Completetest reporting including defects, statistics and characteristics of the test cases
- 56. Coverage • Which coveragemodel to use for mobile GUI testing? • LOC coverage • Widget coverage • Activity coverage • Intent/Transition coverage • State coverage (with models) • Multi-device coverage (Vilkomir, 2018) • …?
- 57. Fragility metrics • Empiricalstudy: modification history of Junit (unit) and Espresso (GUI) tests for a popular open-source Android SUT Coppola, R., Morisio, M., Torchiano, M., & Ardito, L. (2019). Scripted GUI testing of Android open-source apps: evolution of test code and fragility causes. Empirical Software Engineering, 24(5), 3205-3248.
- 58. Fragility metrics • Duringthe evolution of the SUT, keep track of the less «robust» test cases
- 59. Fragility metrics • Objective: •Mark test cases as fragile if they require too much intervention during the evolution of the SUT • Remove from the test suite the layout-based or visual test cases that require too much maintenance • (use other approaches for the involved Activities: random testing, augmented manual…)
- 60.
- 61.
- 62. Test Maintenance • Testsfor new versions of the application should be (at least partly) automatically updated
- 63. Test Maintenance • Testfragilities can be used to select test cases to keep in the test suite
- 64. Test Maintenance • Changesin the app should be intercepted in order to repair broken test cases
- 65. Test Repair • Approachesto test repair: • Based on visual locators and computer Vision • Based on models of the GUI; • Based on widget similarity.
- 66. Test Repair withModels • Available frameworks based on ESMs (Event-Sequence Models)
- 67. Test Repair withModels • A DESM (Delta-ESM) is used to repair test cases
- 68. Test Repair withWidget Similarity Indexes • Formulas to measure the difference between two widgets, to identify widgets that should be treated as the same in presence of changes Hammoudi, M., Rothermel, G., & Stocco, A. (2016, November). Waterfall: An incremental approach for repairing record-replay tests of web applications. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (pp. 751-762).
- 69. Test Repair withWidget Similarity Indexes Widget of v.1 Widget of v.2 x = 1 if w1 and w2 have the same value for attribute a; 0 otherwise Different weights for each attribute Weight normalization
- 70. Test Maintenance • Informationto prioritize test cases: • Bug-finding ability of tests • Bug history of software repository • Usage metrics from users
- 71. Test Prioritization • Nospecific prioritization model for mobile GUI test cases • Existing ones apply to non- functional properties (e.g., security) and to manual test prioritization Sadeghi, A., Esfahani, N., & Malek, S. (2017, September). Mining mobile app markets for prioritization of security assessment effort. In Proceedings of the 2nd ACM SIGSOFT International Workshop on App Market Analytics (pp. 1-7).
- 72.
- 73. What we aredoing • Empirical assessments of benefits of gamification, when applied to web and mobile test suites; • Definition and validation of a Widget Similarity Index for mobile GUI test repair; • Definition of a framework for the integration of visual test cases in continuous integration environments.
- 74. What we planto do • Definition of a mobile-specific coverage scenario; • Definition of test priorization heuristics based on information mined from open-source projects.
- 75. Thank you Contact: riccardo.coppola@polito.it @CardoCoppola http://softeng.polito.itlinkedin.com/in/riccardo-coppola/
Editor's Notes
- #4 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #5 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #6 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #7 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #8 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #9 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #13 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #14 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #15 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #16 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #17 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #18 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #21 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #22 Resources act as mappers between the application protocol exposed by the service and messages to objects representing the domain. Typically, they are thin, with responsibility for sanity checking the request and providing a protocol specific response according to the outcome of the business transaction Almost all of the service logic resides in a domain model representing the business domain. Of these objects, services coordinate across multiple domain activities, whilst repositories act on collections of domain entities and are often persistence backed. If one service has another service as a collaborator, some logic is needed to communicate with the external service. A gateway encapsulates message passing with a remote service Except in the most trivial cases or when a service acts as an aggregator across resources owned by other services, a micro-service will need to be able to persist objects from the domain between requests.
- #34 Obiettivi Performance migliori Maggior coinvolgimento e divertimento Applicazioni Istruzione Ingegneria del Software
- #35 Obiettivi Performance migliori Maggior coinvolgimento e divertimento Applicazioni Istruzione Ingegneria del Software
- #36 Obiettivi Performance migliori Maggior coinvolgimento e divertimento Applicazioni Istruzione Ingegneria del Software
- #37 Obiettivi Performance migliori Maggior coinvolgimento e divertimento Applicazioni Istruzione Ingegneria del Software
- #38 Obiettivi Performance migliori Maggior coinvolgimento e divertimento Applicazioni Istruzione Ingegneria del Software
- #39 Strumento di ludicizzazione dell’attività di mutation testing. Sono presenti due ruoli: Attaccanti: devono introdurre dei bug nel codice per mettere in luce le fragilità della test suite esistente; Difensori: devono irrobustire la test suite arricchendola con test supplementari
- #40 Strumento di ludicizzazione dell’attività di mutation testing. Sono presenti due ruoli: Attaccanti: devono introdurre dei bug nel codice per mettere in luce le fragilità della test suite esistente; Difensori: devono irrobustire la test suite arricchendola con test supplementari
- #67 n this paper, we propose a novel approach, called ATOM, to automatically maintain GUI test scripts of mobile apps for regression testing. ATOM uses an event sequence model to abstract possible event sequences on a GUI and a delta ESM to abstract the changes made to the GUI. Given both models as input, ATOM automatically updates the test scripts written for a base version app to reflect the changes. In an experiment with 22 versions from 11 production Android apps, ATOM updated all the test scripts affected by the version change; the updated scripts achieve over 80% of the coverage by the original scripts on the base version app; all except one set of updated scripts preserve over 60% of the actions in the original test scripts.