The document summarizes challenges in automated GUI testing of Android apps and potential solutions. It discusses essential challenges such as designing test oracles that work across devices and coordinating event timing. It also discusses accidental challenges like ensuring clean application data and dealing with bugs in frameworks and tools. Solutions proposed include using screen hierarchy hashes as oracles, real usage data for external dependencies, clearing app data between tests, and modifying framework components in AOSP.
ICSE17 - Tool Demonstration - CrashScope A Practical Tool for the Automated T...Kevin Moran
This presentation covers the CrashScope tool which uses a combination of static and dynamic analysis to practically and effectively test Android applications. For more information please visit https://www.android-dev-tools.com/crashscope-home/
ICSE17Tech Briefing - Automated GUI Testing of Android Apps: From Research to...Kevin Moran
This is the slide deck for the Technical Briefing given by Kevin Moran, Mario Linares Vasquez, and Denys Poshyvanyk at ICSE17 this year. The presentation covers 1) The basics of GUI Testing and its importance to mobile testing; 2) The current state-of-the-art and practice for testing mobile applications; 3) Current Challenges facing researchers and practitioners aiming to design and build automated testing approaches; and 4) A new direction for automated mobile testing research issued as guidance and a challenge for the SE community.
Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mob...Kevin Moran
Mobile app development involves a unique set of challenges including device fragmentation and rapidly evolving platforms, making testing a difficult task. The design space for a comprehensive mobile testing strategy includes features, inputs, potential contextual app states, and large combinations of devices and underlying platforms. Therefore, automated testing is an essential activity of the development process. However, current state of the art of automated testing tools for mobile apps posses limitations that has driven a preference for manual testing in practice. As of today, there is no comprehensive automated solution for mobile testing that overcomes fundamental issues such as automated oracles, history awareness in test cases, or automated evolution of test cases.
In this perspective paper we survey the current state of the art in terms of the frameworks, tools, and services available to developers to aid in mobile testing, highlighting present shortcom- ings. Next, we provide commentary on current key challenges that restrict the possibility of a comprehensive, effective, and practical automated testing solution. Finally, we offer our vision of a comprehensive mobile app testing framework, complete with research agenda, that is succinctly summarized along three principles: Continuous, Evolutionary and Large-scale (CEL).
Automatically Discovering, Reporting and Reproducing Android Application CrashesKevin Moran
Mobile developers face unique challenges when detecting and reporting crashes in apps due to their prevailing GUI event-driven nature and additional sources of inputs (e.g., sensor readings). To support developers in these tasks, we introduce a novel, automated approach called CRASHSCOPE. This tool explores a given Android app using systematic input generation, according to several strategies informed by static and dynamic analyses, with the intrinsic goal of triggering crashes. When a crash is detected, CRASHSCOPE generates an augmented crash report containing screenshots, detailed crash reproduction steps, the captured exception stack trace, and a fully replayable script that automatically reproduces the crash on a target device(s).
We evaluated CRASHSCOPE’s effectiveness in discovering crashes as compared to five state-of-the-art Android input generation tools on 61 applications. The results demonstrate that CRASHSCOPE performs about as well as current tools for detecting crashes and provides more detailed fault information. Additionally, in a study analyzing eight real-world Android app crashes, we found that CRASHSCOPE’s reports are easily readable and allow for reliable reproduction of crashes by presenting more explicit information than human written reports.
How do Developers Test Android Applications?Kevin Moran
Enabling fully automated testing of mobile applications has recently become an important topic of study for both researchers and practitioners. A plethora of tools and approaches have been proposed to aid mobile developers both by augmenting manual testing practices and by automating various parts of the testing process. However, current approaches for automated testing fall short in convincing developers about their benefits, leading to a majority of mobile testing being performed manually. With the goal of helping researchers and practitioners – who design approaches supporting mobile testing – to understand developer’s needs, we analyzed survey responses from 102 open source contributors to Android projects about their practices when performing testing. The survey focused on questions regarding practices and preferences of developers/testers in-the-wild for (i) designing and generating test cases, (ii) automated testing practices, and (iii) perceptions of quality metrics such as code coverage for determining test quality. Analyzing the information gleaned from this survey, we compile a body of knowledge to help guide researchers and professionals toward tailoring new automated testing approaches to the need of a diverse set of open source developers.
On-Device Bug Reporting for Android ApplicationsKevin Moran
Bugs that surface in mobile applications can be difficult to reproduce and fix due to several confounding factors including the highly GUI-driven nature of mobile apps, varying contextual states, differing platform versions and device fragmentation. It is clear that developers need support in the form of automated tools that allow for more precise reporting of application defects in order to facilitate more efficient and effective bug fixes. In this paper, we present a tool aimed at supporting application testers and developers in the process of On-Device Bug Reporting. Our tool, called ODBR, leverages the uiautomator framework and low-level event stream capture to offer support for recording and replaying a series of input gesture and sensor events that describe a bug in an Android application.
ICSE17 - Tool Demonstration - CrashScope A Practical Tool for the Automated T...Kevin Moran
This presentation covers the CrashScope tool which uses a combination of static and dynamic analysis to practically and effectively test Android applications. For more information please visit https://www.android-dev-tools.com/crashscope-home/
ICSE17Tech Briefing - Automated GUI Testing of Android Apps: From Research to...Kevin Moran
This is the slide deck for the Technical Briefing given by Kevin Moran, Mario Linares Vasquez, and Denys Poshyvanyk at ICSE17 this year. The presentation covers 1) The basics of GUI Testing and its importance to mobile testing; 2) The current state-of-the-art and practice for testing mobile applications; 3) Current Challenges facing researchers and practitioners aiming to design and build automated testing approaches; and 4) A new direction for automated mobile testing research issued as guidance and a challenge for the SE community.
Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mob...Kevin Moran
Mobile app development involves a unique set of challenges including device fragmentation and rapidly evolving platforms, making testing a difficult task. The design space for a comprehensive mobile testing strategy includes features, inputs, potential contextual app states, and large combinations of devices and underlying platforms. Therefore, automated testing is an essential activity of the development process. However, current state of the art of automated testing tools for mobile apps posses limitations that has driven a preference for manual testing in practice. As of today, there is no comprehensive automated solution for mobile testing that overcomes fundamental issues such as automated oracles, history awareness in test cases, or automated evolution of test cases.
In this perspective paper we survey the current state of the art in terms of the frameworks, tools, and services available to developers to aid in mobile testing, highlighting present shortcom- ings. Next, we provide commentary on current key challenges that restrict the possibility of a comprehensive, effective, and practical automated testing solution. Finally, we offer our vision of a comprehensive mobile app testing framework, complete with research agenda, that is succinctly summarized along three principles: Continuous, Evolutionary and Large-scale (CEL).
Automatically Discovering, Reporting and Reproducing Android Application CrashesKevin Moran
Mobile developers face unique challenges when detecting and reporting crashes in apps due to their prevailing GUI event-driven nature and additional sources of inputs (e.g., sensor readings). To support developers in these tasks, we introduce a novel, automated approach called CRASHSCOPE. This tool explores a given Android app using systematic input generation, according to several strategies informed by static and dynamic analyses, with the intrinsic goal of triggering crashes. When a crash is detected, CRASHSCOPE generates an augmented crash report containing screenshots, detailed crash reproduction steps, the captured exception stack trace, and a fully replayable script that automatically reproduces the crash on a target device(s).
We evaluated CRASHSCOPE’s effectiveness in discovering crashes as compared to five state-of-the-art Android input generation tools on 61 applications. The results demonstrate that CRASHSCOPE performs about as well as current tools for detecting crashes and provides more detailed fault information. Additionally, in a study analyzing eight real-world Android app crashes, we found that CRASHSCOPE’s reports are easily readable and allow for reliable reproduction of crashes by presenting more explicit information than human written reports.
How do Developers Test Android Applications?Kevin Moran
Enabling fully automated testing of mobile applications has recently become an important topic of study for both researchers and practitioners. A plethora of tools and approaches have been proposed to aid mobile developers both by augmenting manual testing practices and by automating various parts of the testing process. However, current approaches for automated testing fall short in convincing developers about their benefits, leading to a majority of mobile testing being performed manually. With the goal of helping researchers and practitioners – who design approaches supporting mobile testing – to understand developer’s needs, we analyzed survey responses from 102 open source contributors to Android projects about their practices when performing testing. The survey focused on questions regarding practices and preferences of developers/testers in-the-wild for (i) designing and generating test cases, (ii) automated testing practices, and (iii) perceptions of quality metrics such as code coverage for determining test quality. Analyzing the information gleaned from this survey, we compile a body of knowledge to help guide researchers and professionals toward tailoring new automated testing approaches to the need of a diverse set of open source developers.
On-Device Bug Reporting for Android ApplicationsKevin Moran
Bugs that surface in mobile applications can be difficult to reproduce and fix due to several confounding factors including the highly GUI-driven nature of mobile apps, varying contextual states, differing platform versions and device fragmentation. It is clear that developers need support in the form of automated tools that allow for more precise reporting of application defects in order to facilitate more efficient and effective bug fixes. In this paper, we present a tool aimed at supporting application testers and developers in the process of On-Device Bug Reporting. Our tool, called ODBR, leverages the uiautomator framework and low-level event stream capture to offer support for recording and replaying a series of input gesture and sensor events that describe a bug in an Android application.
Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile AppsKevin Moran
It is common practice for developers of user-facing software to transform a mock-up of a graphical user interface (GUI) into code. This process takes place both at an application’s inception and in an evolutionary context as GUI changes keep pace with evolving features. Unfortunately, this practice is challenging and time-consuming. In this paper, we present an approach that automates this process by enabling accurate prototyping of GUIs via three tasks: detection, classification, and assembly. First, logical components of a GUI are detected from a mock-up artifact using either computer vision techniques or mock-up metadata. Then, software repository mining, automated dynamic analysis, and deep convolutional neural networks are utilized to accurately classify GUI-components into domain-specific types (e.g., toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates a suitable hierarchical GUI structure from which a prototype application can be automatically assembled. We implemented this approach for Android in a system called ReDraw. Our evaluation illustrates that ReDraw achieves an average GUI-component classification accuracy of 91% and assembles prototype applications that closely mirror target mock-ups in terms of visual affinity while exhibiting reasonable code structure. Interviews with industrial practitioners illustrate ReDraw’s potential to improve real development workflows.
Эвристики, мнемоники и другие греческие слова в исследовательском тестировани...SQALab
The document discusses the use of heuristics and mnemonics in mobile application testing. It introduces several techniques for experience-based testing including error guessing, checklist-based testing, and exploratory testing. Several mnemonics and heuristics for mobile testing are presented, including SFDPOT (Structure, Function, Data, Platform, Operations, Time), SFDPOTI (adding Interface), SLICED UP FUN, and the Goldilocks Principle for text lengths. Examples are given for how to use these frameworks to guide testing. References for further reading on the topics are also included.
Software testing tools are evolving. More testing frameworks are emerging through the open source community and commercial vendors. In addition, we’re starting to see the rise of machine-learning (ML) and artificial intelligence (AI) in testing solutions.
Given this evolution, it is important to map the tools that match both the practitioners’ skills and their testing types. When referring to the testing practitioners, we mainly look at three different personas:
-The business tester
-The software developer in test (SDET)
-The software developer
These practitioners are tasked with creating, maintaining, and executing unit tests, build acceptance tests, integration, regression, and other nonfunctional tests.
In this webinar led by Perfecto’s Chief Evangelist, Eran Kinsbruner, you will learn the following:
-How should testing types be dispersed among the three personas and throughout the DevOps pipeline?
-What tools should each of these three personas use for the creation and execution of tests?
-What are the key benefits to continuous testing when mapped correctly?
Rewriting your Master Testing Strategy is in order when you look to tackle your first mobile initiative. This includes getting your test environments ready for testing. Often test environments have been on closed internal networks that are not accessible from the internet. Delivering and testing a mobile application on a 3G or 4G network changes all that. Coming up with a supported platform strategy is also paramount to success. One needs to take network speed, device features, screen size and resolution as well as multi-tasking life-cycles into effect when testing. QA testers will need to become familiar on how to service the devices they test with as well. Additionally the team will need to incorporate field testing prior to delivery. This presentation will organize the challenges a modern QA team has to contend with, and make some strong suggestions on how to craft a respectable Mobile Test Strategy.
Best Practices for DevOps in Mobile App TestingBitbar
Watch a live presentation at http://offer.bitbar.com/best-practices-for-devops-in-mobile-app-testing
In essence, the core of DevOps methodology aims to speed up the app development delivery and process by getting devs and operation specialists to collaborate throughout the end-to-end app development and deployment process.
Stay tuned and join our upcoming webinars at http://bitbar.com/testing/webinars/
The document describes LinkedIn's approach to releasing updates for their flagship mobile app three times per day (dubbed "3x3") in order to enable faster iteration. Key aspects included distributed and parallel testing across devices and emulators, refactoring tests for speed, creating consistent test environments, static analysis of code, and feature flagging to reduce risk and enable A/B testing of changes. The approach helped reduce release cycles from monthly to daily but further automation of areas like production releases and performance testing was still needed. Distributing workload and leveraging services like TestFlight and Google Play helped scale testing across many devices.
Speakers: Chirayu Krishnappa & Daniel Liem (Uber), Christian Legnitto (Pocketship), Jacek Suliga (LinkedIn), Robbert Van Ginkel & Gautam Korlam (Uber), Rachel Brindle (Pivotal Labs)
Tools for building and releasing mobile applications provided by Google and Apple are geared towards smaller mobile application development teams. However, once development teams start to grow, the requirement from your build and release tooling change drastically. This meetup was all about building & releasing mobile applications at scale: How does your build infrastructure need to evolve to support increasingly complex code-bases? How can you build an automated release pipeline that tracks every build and rolls out your mobile applications safely? What kind of unexpected problems can arise from adopting new technologies when you have a larger team?
How to Learn The History of Software Testing Keizo Tatsumi
The document provides a history of software testing covering several topics:
1. It discusses the prehistory of software testing, noting that Ada Lovelace is considered the first programmer and suggesting she may have also been the first tester while working on Charles Babbage's Analytical Engine in the 19th century.
2. It outlines the evolution of computers, software engineering, and the growth of software testing from the 1950s to the present day. Key periods included the debugging, demonstration, destruction, evaluation, and prevention-oriented periods.
3. It describes some of the early testing techniques developed in the 1960s-1970s, including the concept of test control processes at IBM, equivalence partitioning, and boundary
The document discusses using robots for automated testing of Android apps. It describes various automated test input generation techniques like Monkey, ACTEve, Dynodroid, A3E-DF, SwiftHand and GUIRipper. These techniques use different strategies for event exploration like random, model-based or systematic. The document also presents results of experiments comparing these techniques on criteria like ease of use, code coverage achieved and fault detection ability. It finds that while random testing can be effective, no single strategy is sufficient, and a combination of strategies is more effective for testing mobile apps.
For the success of our apps, we often add end user documentation in the form of help notes, tutorials and app demos/tours. Such end user docs are often outdated leading to confused users who make noise on support channels and worse on the play store. Tests also have the same problem — they need to be updated after the app changes to keep them in sync. However, tests produce pass/fail reports to indicate out of sync problems. So, why not kill two birds with one stone? I’ll show how end-to-end UI tests can generate beautiful user docs for happy cases and make sure they're never out of date.
These are the slides for the presentation that Bitbar experts gave at Droidcon Berlin 2012 (http://de.droidcon.com/content/testing-device-compatibility-not-hard-you-think). The presentation included also a live demo of Recording tests with Testdroid Recorder 2.0 and running the same use cases with one click on all 100+ real Android devices in Testdroid Cloud.
Top Best Practices for Successful Mobile Test AutomationFred Beringer
Mobile apps have brought a whole new set of challenges when it comes to testing: Fast paced development cycles with multiple releases per week, multiple app technologies and development platforms to support, tons of devices and form factors, and additional pressure from enterprise and consumers less patient with low quality apps. With these new challenges, come a new set of mistakes testers can make!
Fred has worked with dozens of mobile test teams to help them avoid common traps when building test automation for mobile apps and would like to share some best practices that could be useful to a lot of developers and testers starting with mobile test automation.
Best practices such as:
• When, what and where to automate?
• Picking the right wait to handle unreliable back-end calls or device performance
• Building testability in a mobile app
• Automating the automation (!)
• Mix and match performance testing and functional testing
In this talk, Fred will bring some real stories (struggles!) and how small changes in process made these mobile apps 10x more reliable!
Watch a live presentation at http://offer.bitbar.com/learn-the-best-practices-of-mobile-game-testing
To get the best start for your mobile game - or just to improve the existing game's possibility to get in front of hundreds of millions of gamers - we'll be sharing our insights, best practices and lots of tips&tricks how to gain advantage in this race.
Stay tuned and join our upcoming webinars at http://bitbar.com/testing/webinars/
This document discusses continuous code quality for Java projects. It describes various tools that can be used for static code analysis, including IntelliJ IDEA, PVS Studio, and SonarQube. SonarQube is discussed in depth, including how it can be used to analyze code quality metrics, detect issues, and integrate with CI/CD pipelines. The document also covers other quality practices like code reviews, testing, and ensuring code quality in open source projects.
This document provides an overview and comparison of several popular open source automation tools, including Selenium, Watir, FlexMonkey, TurnkeyLinux, TestLink, and Redmine. Selenium is a suite of tools for automating web browsers across different browsers and operating systems. Watir is a Ruby library for automating web browsers. FlexMonkey is a tool for testing Flex and Air applications. TurnkeyLinux provides over 45 ready-to-use Linux solutions. TestLink is an open source test management tool. Redmine is an open source project management and issue tracking tool. The document compares the pros and cons of each tool and provides example usages and links for further information.
Implementing Test Automation in Agile ProjectsDominik Dary
All new features at eBay Europe are developed using SCRUM. One key success factor for those projects is to have a reliable end-to-end test automation safety net. This presentation illustrates how in addition to a robust automation toolset it is essential to have an integrated approach to test automation design:
Test Aspects - Test Aspects are used to do the functional design of the end-to-end automation test cases. Since this is done upfront, the tester is able to focus on the what rather than the how.
Modeling of the Biz Domain Layer - The Biz Domain Layer is an abstraction layer above the user interface that is implemented in the test code. This layer is divided into pages and flows which are then used in the tests.
Test Implementation - Tests are written in Java, stored in SVN and are executed using the WebDriver Grid (Selenium2). Tests execution can be triggered by all team members via a continuous integration server (Hudson).
Lean Test Automation – it is important to retain and maintain the quality of the tests. Key success factors for this are: Code Reviews, Software Craftsmanship, Test Aspect Reviews and the “Definition of Done”.
Following an integrated approach to test automation ensures high efficiency, low overhead and easier maintenance.
There's a shift to open-source mobile test automation tools happening today among developers and QA. Many mature technology sectors are adopting lightweight, vendor-transparent tools to fulfill the need for speed and integration.
But as with many free and open-source software markets however, a plethora of tools complicates the selection process. How do you know which tool is worth spending the time to learn and integrate in your own environment?
These slides aim to help you choose which open-source test automation framework to use based on a number of critical considerations.
Top 10 Automation Testing Tools in 2020Alaina Carter
Automation testing plays a significant role in building a robust product while enabling Quality at Speed. Using the right automation testing tools at the right time is vital for delivering a quality product. Read more to know what are the top 10 automation testing tools in 2020.
0/5
You’ve built a new feature in your app that is ready ship. Or is it? How can you be sure you’ve not introduced regressions in cases you forgot to test? What if your code crashes only on certain devices? Could the feature freeze up for a few users?
Shipping frequently with little to no functional, UX or performance issues or regressions is not easy - but it’s also a problem that has been solved before. Where things get a lot more interesting is how to keep the same quality bar when you have hundreds of pull requests going in every day, with tens or hundreds of developers working on the same project? How do you test that your app still works - with this kind of scale?
In this talk, you’ll learn about the different approaches we combined into a system used by hundreds of mobile engineers at Uber to test our native iOS and Android apps during development, at release as well as when in production. We’ll talk about what did and what did not work for us on our journey of iterating frequently and continuously improving the quality bar.
Testing Vue Apps with Cypress.io (STLJS Meetup April 2018)Christian Catalan
Presented at the STLJS Meetup (St Louis, MO)
We dive into a Vue application used in semiconductor labs for transistor measurements. We discuss how get started with E2E testing with Cypress.io. And give a crash course into Vue applications.
Video: https://www.youtube.com/watch?v=dpB0YgnFyZQ
Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile AppsKevin Moran
It is common practice for developers of user-facing software to transform a mock-up of a graphical user interface (GUI) into code. This process takes place both at an application’s inception and in an evolutionary context as GUI changes keep pace with evolving features. Unfortunately, this practice is challenging and time-consuming. In this paper, we present an approach that automates this process by enabling accurate prototyping of GUIs via three tasks: detection, classification, and assembly. First, logical components of a GUI are detected from a mock-up artifact using either computer vision techniques or mock-up metadata. Then, software repository mining, automated dynamic analysis, and deep convolutional neural networks are utilized to accurately classify GUI-components into domain-specific types (e.g., toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates a suitable hierarchical GUI structure from which a prototype application can be automatically assembled. We implemented this approach for Android in a system called ReDraw. Our evaluation illustrates that ReDraw achieves an average GUI-component classification accuracy of 91% and assembles prototype applications that closely mirror target mock-ups in terms of visual affinity while exhibiting reasonable code structure. Interviews with industrial practitioners illustrate ReDraw’s potential to improve real development workflows.
Эвристики, мнемоники и другие греческие слова в исследовательском тестировани...SQALab
The document discusses the use of heuristics and mnemonics in mobile application testing. It introduces several techniques for experience-based testing including error guessing, checklist-based testing, and exploratory testing. Several mnemonics and heuristics for mobile testing are presented, including SFDPOT (Structure, Function, Data, Platform, Operations, Time), SFDPOTI (adding Interface), SLICED UP FUN, and the Goldilocks Principle for text lengths. Examples are given for how to use these frameworks to guide testing. References for further reading on the topics are also included.
Software testing tools are evolving. More testing frameworks are emerging through the open source community and commercial vendors. In addition, we’re starting to see the rise of machine-learning (ML) and artificial intelligence (AI) in testing solutions.
Given this evolution, it is important to map the tools that match both the practitioners’ skills and their testing types. When referring to the testing practitioners, we mainly look at three different personas:
-The business tester
-The software developer in test (SDET)
-The software developer
These practitioners are tasked with creating, maintaining, and executing unit tests, build acceptance tests, integration, regression, and other nonfunctional tests.
In this webinar led by Perfecto’s Chief Evangelist, Eran Kinsbruner, you will learn the following:
-How should testing types be dispersed among the three personas and throughout the DevOps pipeline?
-What tools should each of these three personas use for the creation and execution of tests?
-What are the key benefits to continuous testing when mapped correctly?
Rewriting your Master Testing Strategy is in order when you look to tackle your first mobile initiative. This includes getting your test environments ready for testing. Often test environments have been on closed internal networks that are not accessible from the internet. Delivering and testing a mobile application on a 3G or 4G network changes all that. Coming up with a supported platform strategy is also paramount to success. One needs to take network speed, device features, screen size and resolution as well as multi-tasking life-cycles into effect when testing. QA testers will need to become familiar on how to service the devices they test with as well. Additionally the team will need to incorporate field testing prior to delivery. This presentation will organize the challenges a modern QA team has to contend with, and make some strong suggestions on how to craft a respectable Mobile Test Strategy.
Best Practices for DevOps in Mobile App TestingBitbar
Watch a live presentation at http://offer.bitbar.com/best-practices-for-devops-in-mobile-app-testing
In essence, the core of DevOps methodology aims to speed up the app development delivery and process by getting devs and operation specialists to collaborate throughout the end-to-end app development and deployment process.
Stay tuned and join our upcoming webinars at http://bitbar.com/testing/webinars/
The document describes LinkedIn's approach to releasing updates for their flagship mobile app three times per day (dubbed "3x3") in order to enable faster iteration. Key aspects included distributed and parallel testing across devices and emulators, refactoring tests for speed, creating consistent test environments, static analysis of code, and feature flagging to reduce risk and enable A/B testing of changes. The approach helped reduce release cycles from monthly to daily but further automation of areas like production releases and performance testing was still needed. Distributing workload and leveraging services like TestFlight and Google Play helped scale testing across many devices.
Speakers: Chirayu Krishnappa & Daniel Liem (Uber), Christian Legnitto (Pocketship), Jacek Suliga (LinkedIn), Robbert Van Ginkel & Gautam Korlam (Uber), Rachel Brindle (Pivotal Labs)
Tools for building and releasing mobile applications provided by Google and Apple are geared towards smaller mobile application development teams. However, once development teams start to grow, the requirement from your build and release tooling change drastically. This meetup was all about building & releasing mobile applications at scale: How does your build infrastructure need to evolve to support increasingly complex code-bases? How can you build an automated release pipeline that tracks every build and rolls out your mobile applications safely? What kind of unexpected problems can arise from adopting new technologies when you have a larger team?
How to Learn The History of Software Testing Keizo Tatsumi
The document provides a history of software testing covering several topics:
1. It discusses the prehistory of software testing, noting that Ada Lovelace is considered the first programmer and suggesting she may have also been the first tester while working on Charles Babbage's Analytical Engine in the 19th century.
2. It outlines the evolution of computers, software engineering, and the growth of software testing from the 1950s to the present day. Key periods included the debugging, demonstration, destruction, evaluation, and prevention-oriented periods.
3. It describes some of the early testing techniques developed in the 1960s-1970s, including the concept of test control processes at IBM, equivalence partitioning, and boundary
The document discusses using robots for automated testing of Android apps. It describes various automated test input generation techniques like Monkey, ACTEve, Dynodroid, A3E-DF, SwiftHand and GUIRipper. These techniques use different strategies for event exploration like random, model-based or systematic. The document also presents results of experiments comparing these techniques on criteria like ease of use, code coverage achieved and fault detection ability. It finds that while random testing can be effective, no single strategy is sufficient, and a combination of strategies is more effective for testing mobile apps.
For the success of our apps, we often add end user documentation in the form of help notes, tutorials and app demos/tours. Such end user docs are often outdated leading to confused users who make noise on support channels and worse on the play store. Tests also have the same problem — they need to be updated after the app changes to keep them in sync. However, tests produce pass/fail reports to indicate out of sync problems. So, why not kill two birds with one stone? I’ll show how end-to-end UI tests can generate beautiful user docs for happy cases and make sure they're never out of date.
These are the slides for the presentation that Bitbar experts gave at Droidcon Berlin 2012 (http://de.droidcon.com/content/testing-device-compatibility-not-hard-you-think). The presentation included also a live demo of Recording tests with Testdroid Recorder 2.0 and running the same use cases with one click on all 100+ real Android devices in Testdroid Cloud.
Top Best Practices for Successful Mobile Test AutomationFred Beringer
Mobile apps have brought a whole new set of challenges when it comes to testing: Fast paced development cycles with multiple releases per week, multiple app technologies and development platforms to support, tons of devices and form factors, and additional pressure from enterprise and consumers less patient with low quality apps. With these new challenges, come a new set of mistakes testers can make!
Fred has worked with dozens of mobile test teams to help them avoid common traps when building test automation for mobile apps and would like to share some best practices that could be useful to a lot of developers and testers starting with mobile test automation.
Best practices such as:
• When, what and where to automate?
• Picking the right wait to handle unreliable back-end calls or device performance
• Building testability in a mobile app
• Automating the automation (!)
• Mix and match performance testing and functional testing
In this talk, Fred will bring some real stories (struggles!) and how small changes in process made these mobile apps 10x more reliable!
Watch a live presentation at http://offer.bitbar.com/learn-the-best-practices-of-mobile-game-testing
To get the best start for your mobile game - or just to improve the existing game's possibility to get in front of hundreds of millions of gamers - we'll be sharing our insights, best practices and lots of tips&tricks how to gain advantage in this race.
Stay tuned and join our upcoming webinars at http://bitbar.com/testing/webinars/
This document discusses continuous code quality for Java projects. It describes various tools that can be used for static code analysis, including IntelliJ IDEA, PVS Studio, and SonarQube. SonarQube is discussed in depth, including how it can be used to analyze code quality metrics, detect issues, and integrate with CI/CD pipelines. The document also covers other quality practices like code reviews, testing, and ensuring code quality in open source projects.
This document provides an overview and comparison of several popular open source automation tools, including Selenium, Watir, FlexMonkey, TurnkeyLinux, TestLink, and Redmine. Selenium is a suite of tools for automating web browsers across different browsers and operating systems. Watir is a Ruby library for automating web browsers. FlexMonkey is a tool for testing Flex and Air applications. TurnkeyLinux provides over 45 ready-to-use Linux solutions. TestLink is an open source test management tool. Redmine is an open source project management and issue tracking tool. The document compares the pros and cons of each tool and provides example usages and links for further information.
Implementing Test Automation in Agile ProjectsDominik Dary
All new features at eBay Europe are developed using SCRUM. One key success factor for those projects is to have a reliable end-to-end test automation safety net. This presentation illustrates how in addition to a robust automation toolset it is essential to have an integrated approach to test automation design:
Test Aspects - Test Aspects are used to do the functional design of the end-to-end automation test cases. Since this is done upfront, the tester is able to focus on the what rather than the how.
Modeling of the Biz Domain Layer - The Biz Domain Layer is an abstraction layer above the user interface that is implemented in the test code. This layer is divided into pages and flows which are then used in the tests.
Test Implementation - Tests are written in Java, stored in SVN and are executed using the WebDriver Grid (Selenium2). Tests execution can be triggered by all team members via a continuous integration server (Hudson).
Lean Test Automation – it is important to retain and maintain the quality of the tests. Key success factors for this are: Code Reviews, Software Craftsmanship, Test Aspect Reviews and the “Definition of Done”.
Following an integrated approach to test automation ensures high efficiency, low overhead and easier maintenance.
There's a shift to open-source mobile test automation tools happening today among developers and QA. Many mature technology sectors are adopting lightweight, vendor-transparent tools to fulfill the need for speed and integration.
But as with many free and open-source software markets however, a plethora of tools complicates the selection process. How do you know which tool is worth spending the time to learn and integrate in your own environment?
These slides aim to help you choose which open-source test automation framework to use based on a number of critical considerations.
Top 10 Automation Testing Tools in 2020Alaina Carter
Automation testing plays a significant role in building a robust product while enabling Quality at Speed. Using the right automation testing tools at the right time is vital for delivering a quality product. Read more to know what are the top 10 automation testing tools in 2020.
0/5
You’ve built a new feature in your app that is ready ship. Or is it? How can you be sure you’ve not introduced regressions in cases you forgot to test? What if your code crashes only on certain devices? Could the feature freeze up for a few users?
Shipping frequently with little to no functional, UX or performance issues or regressions is not easy - but it’s also a problem that has been solved before. Where things get a lot more interesting is how to keep the same quality bar when you have hundreds of pull requests going in every day, with tens or hundreds of developers working on the same project? How do you test that your app still works - with this kind of scale?
In this talk, you’ll learn about the different approaches we combined into a system used by hundreds of mobile engineers at Uber to test our native iOS and Android apps during development, at release as well as when in production. We’ll talk about what did and what did not work for us on our journey of iterating frequently and continuously improving the quality bar.
Testing Vue Apps with Cypress.io (STLJS Meetup April 2018)Christian Catalan
Presented at the STLJS Meetup (St Louis, MO)
We dive into a Vue application used in semiconductor labs for transistor measurements. We discuss how get started with E2E testing with Cypress.io. And give a crash course into Vue applications.
Video: https://www.youtube.com/watch?v=dpB0YgnFyZQ
Антон Семенченко | (EPAM Systems, DPI.Solutions )Сравнительный анализ инстру...RIF-Technology
Доклад будет посвящен выбору инструментов Автоматизации Desktop-ных приложений. Какие критерии (22) выбора инструмента существуют, подмножество наиболее важных и интересных заказчику, «исполнителю», менеджеру. Какие инструменты (16) представлены сегодня на рынке? Сравнительные характеристики каждого из инструментов по 22 критериям. Также затронем тренды в Автоматизации тестирования Desktop-ных приложений.
This document discusses test automation for mobile applications. It notes challenges with testing across different devices and OS versions as well as the repetitive nature of regression testing. Automation can help address these issues by running tests faster, repeatedly, and comprehensively compared to manual testing. The document reviews options for test automation frameworks including Frank, KIF, and UIAutomation and their pros and cons. It provides guidance on preparing an application for testing by adding accessibility info. Finally, it covers writing, running, and debugging automation scripts.
Talk: https://youtu.be/P1149dWnl3k
Presentation of μ/log and the motivation behind the need of yet, another logging system.
We will cover the main features of the library and what's coming next.
The document discusses automating exploratory testing by creating an app crawler that can emulate user interactions, test across different platforms, languages, resolutions and orientations. Some key points:
- Automated exploratory testing is needed due to shorter release cycles and less manual testing time.
- The author created an app crawler that can capture elements, images, errors, exceptions, performance data and replay tests across platforms.
- Challenges include handling authentication, detecting unique locators, and rescuing apps that get stuck.
- The crawler detects languages, monitors logs and exceptions, and uses Applitools for image validation to further automate the process.
- The goal is to help fill quality gaps
My talk at CodeFest 2017 in Novosibirsk, Russia. I talk about the benefits of adding a app crawler to your build process. In todays Agile world it's becoming difficult to keep up with the amount of manual and exploratory testing with shorter and shorter sprint iterations. It's time to put machines to work and help take some of the load off of us!
Software Testing: Application And Script Independent Automation Framework: Th...guest0efb5e
Raj Kamal from Microsoft India presented on the need for data normalization in test automation frameworks. He discussed common problems with current automation approaches like redundancy, inconsistency, and inability to adapt to changing development models. He proposed a solution called the Application & Script Independent Framework (ASIF) that uses data normalization, user-defined views, and a reusable library to simplify scripts and make them resilient to changes. ASIF provides benefits like reduced maintenance costs and faster turnaround for requirement changes while achieving a 72% return on investment.
Slide deck from the talk I gave at the conference
1. QuickView for test data-> quick feedback
2. Automating everything on top of open-source tools like Selenium, Appium, jenkins etc.
3. Pain points and work arounds specially for Mobile App Test Automation
We are using a variety of automation tools in our project. Appium is one of them. Since Appium is now one of the most used open source automation tools for Mobile we like to share our 2 years of experience: When do we use Appium? How do we use it? What does our architecture look like? Where are the pitfalls? Simply put: How does Appium fare in our day to day life and would we use it again?
Good practices for debugging Selenium and Appium testsAbhijeet Vaikar
We often come across situations when:
> We cannot figure out why our Selenium/Appium scripts fail during execution either locally or on CI.
> We need to debug scripts locally while writing them but find debugging painful and slow
Debugging the scripts is often the approach to fix them. What all different ways we can do it?
I shared about some of the good practices I have used for debugging Selenium and Appium scripts
This document discusses best practices for automated testing on iOS using XCUITest. XCUITest is the native UI testing framework for iOS that supports both Swift and Objective-C. The document recommends automating 5% of tests at the unit level and 95% at integration, functional, and UI testing levels. It provides guidance on testing with SDK/API integrations, mocking network layers, and achieving reliable, maintainable test code through practices like continuous testing, avoiding flaky tests, and optimizing page objects.
Meticulous Planning of Test Automationssuser2e8d4b
The document discusses test automation frameworks, applications, and challenges. It introduces topics around revisiting test automation approaches, evaluating the need for AI/ML in testing, ensuring a precise testing approach, and comparing licensed versus open source test automation solutions. The presentation aims to have an open discussion around improving test automation through meticulous planning.
Test Automation Demonstration with Dr Yongyan Wang by XBOSoftXBOSoft
Dr. Yongyan Wang presented on a test automation framework (TAF). Some key points:
- TAF is an open source framework for automating repeatable tasks across different languages/tools. It uses a distributed architecture.
- Challenges addressed include making automation independent of test/tool types and ensuring consistency. TAF's design focuses on repeatability, adaptability, and a simple web interface.
- A demonstration showed how TAF can simplify test management and provide pass/fail reports. It also integrates other tools like AutoIt and allows command-line usage.
- TAF aims to make automation easy, reliable, reusable, and independent through its structured approach and extens
Experitest and Tech Mahindra on joint webinar on the topic :"UFT vs Selenium- which is better for your mobile strategy?"
This session address the issues mobile testers face when deciding which testing environment to use:
- Should I migrate from UFT to Selenium?
- How important is a user interface?
- How much programming experience do you require in your testing team?
- One test at a time or all of them at once?
- How vital is third party tech support?
- Can I get the best of both?
At Experitest we understand that the best option is to optimize the benefits of both choices. Our SeeTest UFT Addon enables you to extend your UFT testing to mobile applications. We also support Selenium, enabling you to create test scripts that run on this framework.
Watch this webinar to learn how TechM & Experitest can help you succeed choosing the right testing environment for your needs.
The Present and Future of Mobile Test Automation with AppiumTechWell
Mobile testing has moved forward in leaps and bounds in the past few years. With frameworks like Appium, it has never been easier to scale automated testing and integrate it with continuous integration. While new automation technologies proliferate with device and platform upgrades, Appium provides a singular, consistent automation protocol on top of the best automation technologies,. In the past year, the Appium open source community has added support for new kinds of automation, including XCUITest for iOS, UiAutomator 2 for Android, and even Windows apps. With demos and code samples, Jonathan Lipps explores the capabilities these unlock, and explains Appium's vision for the future—in both mobile testing and the rich variety of UI testing with IoT applications and devices. More and more, the lines are being blurred between desktop, mobile, and IoT applications—and Appium has a plan for taking advantage of this remarkable confluence.
Trainmesofttech is well known for its Manual Testing training in Hyderabad. Trainmesofttech provides quality education in training its students. Trainers at trainmesofttech are very well experienced in both training and real-time scenario. Trainmesofttech deals with all types of software training, specialized in testing areas like Manual Testing, Software Testing, QTP Testing, Selenium Testing and many more.
The document discusses mobile end-to-end testing and exploratory testing. It describes two categories of mobile automation techniques: instrumented and non-instrumented. It then covers the advantages of each technique, considerations for which to use, examples of mobile automation tools for Android and iOS, and outlines "walls of pain" related to testing on these platforms.
Similar to Automated GUI-Testing of Android Apps: From Research to Practice (20)
Must Know Postgres Extension for DBA and Developer during MigrationMydbops
Mydbops Opensource Database Meetup 16
Topic: Must-Know PostgreSQL Extensions for Developers and DBAs During Migration
Speaker: Deepak Mahto, Founder of DataCloudGaze Consulting
Date & Time: 8th June | 10 AM - 1 PM IST
Venue: Bangalore International Centre, Bangalore
Abstract: Discover how PostgreSQL extensions can be your secret weapon! This talk explores how key extensions enhance database capabilities and streamline the migration process for users moving from other relational databases like Oracle.
Key Takeaways:
* Learn about crucial extensions like oracle_fdw, pgtt, and pg_audit that ease migration complexities.
* Gain valuable strategies for implementing these extensions in PostgreSQL to achieve license freedom.
* Discover how these key extensions can empower both developers and DBAs during the migration process.
* Don't miss this chance to gain practical knowledge from an industry expert and stay updated on the latest open-source database trends.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: info@mydbops.com
Visit: https://www.mydbops.com/
Follow us on LinkedIn: https://in.linkedin.com/company/mydbops
For more details and updates, please follow up the below links.
Meetup Page : https://www.meetup.com/mydbops-databa...
Twitter: https://twitter.com/mydbopsofficial
Blogs: https://www.mydbops.com/blog/
Facebook(Meta): https://www.facebook.com/mydbops/
Discover the Unseen: Tailored Recommendation of Unwatched ContentScyllaDB
The session shares how JioCinema approaches ""watch discounting."" This capability ensures that if a user watched a certain amount of a show/movie, the platform no longer recommends that particular content to the user. Flawless operation of this feature promotes the discover of new content, improving the overall user experience.
JioCinema is an Indian over-the-top media streaming service owned by Viacom18.
In our second session, we shall learn all about the main features and fundamentals of UiPath Studio that enable us to use the building blocks for any automation project.
📕 Detailed agenda:
Variables and Datatypes
Workflow Layouts
Arguments
Control Flows and Loops
Conditional Statements
💻 Extra training through UiPath Academy:
Variables, Constants, and Arguments in Studio
Control Flow in Studio
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsDianaGray10
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...Jason Yip
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...DanBrown980551
This LF Energy webinar took place June 20, 2024. It featured:
-Alex Thornton, LF Energy
-Hallie Cramer, Google
-Daniel Roesler, UtilityAPI
-Henry Richardson, WattTime
In response to the urgency and scale required to effectively address climate change, open source solutions offer significant potential for driving innovation and progress. Currently, there is a growing demand for standardization and interoperability in energy data and modeling. Open source standards and specifications within the energy sector can also alleviate challenges associated with data fragmentation, transparency, and accessibility. At the same time, it is crucial to consider privacy and security concerns throughout the development of open source platforms.
This webinar will delve into the motivations behind establishing LF Energy’s Carbon Data Specification Consortium. It will provide an overview of the draft specifications and the ongoing progress made by the respective working groups.
Three primary specifications will be discussed:
-Discovery and client registration, emphasizing transparent processes and secure and private access
-Customer data, centering around customer tariffs, bills, energy usage, and full consumption disclosure
-Power systems data, focusing on grid data, inclusive of transmission and distribution networks, generation, intergrid power flows, and market settlement data
"NATO Hackathon Winner: AI-Powered Drug Search", Taras KlobaFwdays
This is a session that details how PostgreSQL's features and Azure AI Services can be effectively used to significantly enhance the search functionality in any application.
In this session, we'll share insights on how we used PostgreSQL to facilitate precise searches across multiple fields in our mobile application. The techniques include using LIKE and ILIKE operators and integrating a trigram-based search to handle potential misspellings, thereby increasing the search accuracy.
We'll also discuss how the azure_ai extension on PostgreSQL databases in Azure and Azure AI Services were utilized to create vectors from user input, a feature beneficial when users wish to find specific items based on text prompts. While our application's case study involves a drug search, the techniques and principles shared in this session can be adapted to improve search functionality in a wide range of applications. Join us to learn how PostgreSQL and Azure AI can be harnessed to enhance your application's search capability.
QA or the Highway - Component Testing: Bridging the gap between frontend appl...zjhamm304
These are the slides for the presentation, "Component Testing: Bridging the gap between frontend applications" that was presented at QA or the Highway 2024 in Columbus, OH by Zachary Hamm.
This talk will cover ScyllaDB Architecture from the cluster-level view and zoom in on data distribution and internal node architecture. In the process, we will learn the secret sauce used to get ScyllaDB's high availability and superior performance. We will also touch on the upcoming changes to ScyllaDB architecture, moving to strongly consistent metadata and tablets.
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...Fwdays
Direct losses from downtime in 1 minute = $5-$10 thousand dollars. Reputation is priceless.
As part of the talk, we will consider the architectural strategies necessary for the development of highly loaded fintech solutions. We will focus on using queues and streaming to efficiently work and manage large amounts of data in real-time and to minimize latency.
We will focus special attention on the architectural patterns used in the design of the fintech system, microservices and event-driven architecture, which ensure scalability, fault tolerance, and consistency of the entire system.
Essentials of Automations: Exploring Attributes & Automation ParametersSafe Software
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
What is an RPA CoE? Session 2 – CoE RolesDianaGray10
In this session, we will review the players involved in the CoE and how each role impacts opportunities.
Topics covered:
• What roles are essential?
• What place in the automation journey does each role play?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
Session 1 - Intro to Robotic Process Automation.pdfUiPathCommunity
👉 Check out our full 'Africa Series - Automation Student Developers (EN)' page to register for the full program:
https://bit.ly/Automation_Student_Kickstart
In this session, we shall introduce you to the world of automation, the UiPath Platform, and guide you on how to install and setup UiPath Studio on your Windows PC.
📕 Detailed agenda:
What is RPA? Benefits of RPA?
RPA Applications
The UiPath End-to-End Automation Platform
UiPath Studio CE Installation and Setup
💻 Extra training through UiPath Academy:
Introduction to Automation
UiPath Business Automation Platform
Explore automation development with UiPath Studio
👉 Register here for our upcoming Session 2 on June 20: Introduction to UiPath Studio Fundamentals: https://community.uipath.com/events/details/uipath-lagos-presents-session-2-introduction-to-uipath-studio-fundamentals/
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor IvaniukFwdays
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
"Scaling RAG Applications to serve millions of users", Kevin GoedeckeFwdays
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Automated GUI-Testing of Android Apps: From Research to Practice
1. ICSME16
Raleigh, NC
Wednesday, October 5th, 2016
Kevin Moran,
Mario Linares-Vásquez,
& Denys Poshyvanyk
College of William & Mary - SEMERU - Department of Computer Science
Automated GUITesting of Android
Apps: From Research to Practice
&
15. Mobile App Dev Challenges
• Rapid platform (and third-party lib) evolution
• API-instability and fault-proneness
• Influx of unstructured (“low quality”) user reviews
• Platform-fragmentation at device and OS level
• Quality attributes (performance, energy) and
inputs (sensors, camera, WiFi etc.)
17. Study: mining of code changes
history (5,848 apps); survey with 45
developers
Method: descriptive statistics,
statistical hypothesis testing, and
open coding
Main findings:
- Successful apps use more stable and bug
free APIs
- APIs issues are perceived by users as app
programming issues
- Lack of tools for identifying API issues and
their impact on a new API release
Android (andThird Party library) Instability
19. Mobile AppTesting Challenges
• Highly event-driven, gesture based nature of apps
• Lack of effective automated input generation
• Lack of robust (semi) automated testing and
monitoring tools
• Lack of support for debugging apps and bug
reporting
Joorabchi, M.E.; Mesbah, A.; Kruchten, P., "Real Challenges in Mobile App Development," in Empirical Software
Engineering and Measurement, 2013 ACM / IEEE International Symposium on , vol., no., pp.15-24, 10-11 Oct. 2013
40. Pros and Cons
R&R ✓Easy reproduction - Expensive collection
and maintenance
- Coupled to locations
MBT ✓ Event sequences
✓ Automatic exploration
- Invalid sequences (a-
priori model)
- Few corner cases
- Use cases are not
considered
- Manual execution
- Incomplete models
41. Pros and Cons
R&R ✓Easy reproduction - Expensive collection
and maintenance
- Coupled to locations
MBT ✓ Event sequences
✓ Automatic exploration
- Invalid sequences (a-
priori model)
- Few corner cases
- Use cases are not
considered
- Manual execution
- Incomplete models
Rippers
- Lack of context
- No history awareness
45. Pros and Cons
R&R ✓ Easy reproduction - Expensive collection
and maintenance
- Coupled to locations
MBT ✓ Event sequences
✓ Automatic exploration
- Invalid sequences (a-
priori model)
- Few corner cases
- Use cases are not
considered
- Manual execution
- Incomplete models
Rippers
- Lack of context
- No history awareness
R/FT ✓ Fast execution
✓ Good at finding crashes
- Invalid events
- Lack of expressiveness
48. Pros and Cons
R&R ✓ Easy reproduction
- Expensive collection
and maintenance
- Coupled to locations
MBT ✓ Event sequences
✓ Automatic exploration
- Invalid sequences (a-priori
model)
- Few corner cases
- Use cases are not considered
- Manual execution
- Incomplete models
Rippers
- Lack of context
- No history awareness
R/FT ✓ Fast execution
✓ Good at finding crashes
- Invalid events
- Lack of expressiveness
AF/A ✓ Easy reproduction
✓ High level syntax
✓ Black box testing
- Learning curve
- User-defined oracles
- Expensive maintenance
49. RESEARCH PROTOTYPES
Tool Name Instrumentation GUI Exploration Types of Events
Replayable Test
Cases
Emulators, Devices
Dynodroid Yes Guided/Random System, GUI, Text No No
EvoDroid No System/Evo GUI No N/A
AndroidRipper Yes Systematic GUI, Text No N/A
MobiGUItar Yes Model-Based GUI, Text Yes N/A
A3E Depth-First Yes Systematic GUI No Yes
A3E Targeted Yes Model-Based GUI No Yes
Swifthand Yes Model-Based GUI, Text No Yes
PUMA Yes Programmable System, GUI, Text No Yes
ACTEve Yes Systematic GUI No Yes
VANARSena Yes Random System, GUI, Text Yes N/A
Thor Yes Test Cases Test Case Events N/A No
QUANTUM Yes Model-Based System, GUI Yes N/A
AppDoctor Yes Multiple System, GUI2, Text Yes N/A
ORBIT No Model-Based GUI No N/A
SPAG-C No Record/Replay GUI N/A No
JPF-Android No Scripting GUI Yes N/A
MonkeyLab No Model-based GUI, Text Yes Yes
CrashDroid No Manual Rec/Replay GUI, Text Yes Yes
SIG-Droid No Symbolic GUI, Text Yes N/A
CrashScope No Systematic GUI, Text, System Yes Yes
Sapienz Yes Evolutionary GUI, Text,System Yes Yes
55. Tools: Summary
Testing Automation Frameworks/APIs
Crowdsourced Testing/Cloud-based Testing
Record and Replay
Monkey (Fuzz testing) ✓ Unexpected scenarios
- Reproducibility/Expressiveness
56. Tools: Summary
Testing Automation Frameworks/APIs
Crowdsourced Testing/Cloud-based Testing
Record and Replay
✓ Easy reproduction
✓ Automation on different devices
- Test scripts maintenance
- Learning curve
Monkey (Fuzz testing) ✓ Unexpected scenarios
- Reproducibility/Expressiveness
57. Tools: Summary
Testing Automation Frameworks/APIs
Crowdsourced Testing/Cloud-based Testing
Record and Replay
✓ Automation on different devices
✓ Similar to manual testing
- Test scripts maintenance
✓ Easy reproduction
✓ Automation on different devices
- Test scripts maintenance
- Learning curve
Monkey (Fuzz testing) ✓ Unexpected scenarios
- Reproducibility/Expressiveness
58. Tools: Summary
Testing Automation Frameworks/APIs
Crowdsourced Testing/Cloud-based Testing
Record and Replay
- Open Source vs Paid
✓ Large set of devices
✓ Remote data collection
✓ Support automation frameworks
✓ Automation on different devices
✓ Similar to manual testing
- Test scripts maintenance
✓ Easy reproduction
✓ Automation on different devices
- Test scripts maintenance
- Learning curve
Monkey (Fuzz testing) ✓ Unexpected scenarios
- Reproducibility/Expressiveness
60. Supporting evolution and
maintenance of
Android apps
Mining Android
Software
Repositories
Android API instability
[FSE’13][TSE’15]
Performance
bottlenecks [ICSME’15]
[MSR’16]
User reviews [ICSME’15]
Energy [MSR’14]
61. Supporting evolution and
maintenance of
Android apps
Mining Android
Software
Repositories
Android API instability
[FSE’13][TSE’15]
Performance
bottlenecks [ICSME’15]
[MSR’16]
User reviews [ICSME’15]
Energy [MSR’14]
Crowdsource-based
requirements
[ICSME’15]
GUI - based testing
[MSR’15]
Bug reporting [FSE’15]
Optimizing energy
consumption [FSE’15]
62. SEMERUTools: Monkey Lab
• Testing approach based on natural usages of
an application.
• Models usage using n-gram language models to
produce unseen yet natural event sequences.
• Capable of generating orthogonal test cases
to those already modeled.
63. SEMERUTools: Monkey Lab
• Testing approach based on natural usages of
an application.
• Models usage using n-gram language models to
produce unseen yet natural event sequences.
• Capable of generating orthogonal test cases
to those already modeled.
64. SEMERUTools: FUSION
www.fusion-android.com
• Novel Bug Reporting System for GUI-based
and mobile apps
• Uses a combination of static and dynamic
analysis to help inform the reporting process
• Uses auto-completion to ease the burden on
reporters and ensure high-quality reports.
65. SEMERUTools: FUSION
www.fusion-android.com
• Novel Bug Reporting System for GUI-based
and mobile apps
• Uses a combination of static and dynamic
analysis to help inform the reporting process
• Uses auto-completion to ease the burden on
reporters and ensure high-quality reports.
66. SEMERUTools: CRASHSCOPE
• Completely automated testing approach for Android apps
• Generates detailed, expressive bug reports and repayable
scripts
• A practical tool, requiring no instrumentation framework, or
modification to the OS or applications
• Capable of running on both physical devices and emulators
• Differing execution strategies able to test contextual features
67. SEMERUTools: CRASHSCOPE
• Completely automated testing approach for Android apps
• Generates detailed, expressive bug reports and repayable
scripts
• A practical tool, requiring no instrumentation framework, or
modification to the OS or applications
• Capable of running on both physical devices and emulators
• Differing execution strategies able to test contextual features
68. SEMERUTools: CRASHSCOPE
• Completely automated testing approach for Android apps
• Generates detailed, expressive bug reports and repayable
scripts
• A practical tool, requiring no instrumentation framework, or
modification to the OS or applications
• Capable of running on both physical devices and emulators
• Differing execution strategies able to test contextual features
71. Essential Challenges in Mobile GUITesting
Essential Challenges
Test Oracles: How does one design an extensible GUI- based
oracle that can be used to verify expected application behavior
across different device configurations?
Event Coordination: In automated GUI-based testing, deciding the
proper timing of firing events on a device is challenging.
External Features: How can external features such as notifications
be properly incorporated into large-scale testing of mobile apps?
External Dependencies: Many mobile apps use external servers or
databases in order to send, store, and receive information. How can
these dependencies be appropriately incorporated into large-scale
testing?
72. Essential Challenges in Mobile GUITesting
Essential Challenges
Test Oracles: How does one design an extensible GUI- based
oracle that can be used to verify expected application behavior
across different device configurations?
Event Coordination: In automated GUI-based testing, deciding the
proper timing of firing events on a device is challenging.
External Features: How can external features such as notifications
be properly incorporated into large-scale testing of mobile apps?
External Dependencies: Many mobile apps use external servers or
databases in order to send, store, and receive information. How can
these dependencies be appropriately incorporated into large-scale
testing?
73. Accidental Challenges in Mobile GUITesting
Accidental Challenges
Application Data and Cold Starts: How can one ensure a sterile or
desired testing environment within the context of large scale
testing?
Bugs in Framework Utilities: There exist certain limitations with
core Android Frameworks popular with GUI-testing.
Bugs and Limitations in the SDK Tools: There are limitations
inherent to Android developer tools, such as adb, that make
large-scale parallel execution and testing difficult.
Challenges Scaling Concurrent Virtual Devices: Android emulators
were not designed to scale well on modest hardware.
74. Accidental Challenges in Mobile GUITesting
Accidental Challenges
Application Data and Cold Starts: How can one ensure a sterile or
desired testing environment within the context of large scale
testing?
Bugs in Framework Utilities: There exist certain limitations with
core Android Frameworks popular with GUI-testing.
Bugs and Limitations in the SDK Tools: There are limitations
inherent to Android developer tools, such as adb, that make
large-scale parallel execution and testing difficult.
Challenges Scaling Concurrent Virtual Devices: Android emulators
were not designed to scale well on modest hardware.
75. Essential Challenges:Test Oracles
Potential Solution: Use a hash of the current screen hierarchy
Drawbacks: May miss some fined grained information about the
GUI (e.g. colors, images)
76. Essential Challenges:Test Oracles
Potential Solution: Use a hash of the current screen hierarchy
adb shell shell /system/bin/uiautomator
dump /sdcard/ui_dump.xml
Drawbacks: May miss some fined grained information about the
GUI (e.g. colors, images)
94. Enabling Large Scale Analysis: Execution Engine
✓Support for multiple
maintenance tasks
✓Open/Closed principle
✓Horizontal/Vertical scalability
✓Asynchronous communication
between clients and execution
engine
✓Support for fragmentation
Message bus
Execution Engine
Virtual devices (Pool)
95. Enabling Large Scale Analysis: Execution Engine
Dispatcher
Worker
Task
Device/VM/
Thread
token
Task, Device/VM/
Thread token
Task status,
response
Task status
- Task-specific
frameworks
- Third party libraries
<<invokes>>
Client i
Message Bus
Local DB
Client-specific
technologies
Request
(i.e., new Task)
Web folder or
Content Delivery
Network
Images,
files, etc.Images,
files, etc.
Response (i.e.,
Task results
and status)
Execution Engine (EE)
EE Settings
Devices/VMs
pool
Background
Threads pool
Clients
Client n
96. Enabling Large Scale Analysis: Execution Engine
Dispatcher
Worker
Task
Device/VM/
Thread
token
Task, Device/VM/
Thread token
Task status,
response
Task status
- Task-specific
frameworks
- Third party libraries
<<invokes>>
Client i
Message Bus
Local DB
Client-specific
technologies
Request
(i.e., new Task)
Web folder or
Content Delivery
Network
Images,
files, etc.Images,
files, etc.
Response (i.e.,
Task results
and status)
Execution Engine (EE)
EE Settings
Devices/VMs
pool
Background
Threads pool
Clients
Client n