Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
ICSME16
Raleigh, NC
Wednesday, October 5th, 2016
Kevin Moran,
Mario Linares-Vásquez,
& Denys Poshyvanyk
College of William...
GUI-Based Software Testing
UI Events
GUITesting: The Main Idea
UI Events
Oracle
GUITesting: The Main Idea
Output, layout, exceptions,
presentation logic, quality attributes, …
UI Events
Oracle
GUITesting: The Main Idea
GUITesting: Core Concepts
Test
Result
Oracle
GUITesting: Core Concepts
Test
Result
Oracle
Monkey Testing !!
Monkey Testing !!
Automated Testing !!
Automated GUI-Testing
Output, layout, exceptions,
presentation logic, quality attributes, …
UI Events
Monkey
GUI-Based Testing for Mobile Apps
The App Economy
http://de.my-walls.net/android-os-robot/
10
10
Mobile App Dev Challenges
• Rapid platform (and third-party lib) evolution
• API-instability and fault-proneness
• Influx o...
Android (andThird Party library) Instability
Study: mining of code changes
history (5,848 apps); survey with 45
developers
Method: descriptive statistics,
statistical ...
LargeVolumed of Crowdsourced
Requirements and Ratings
Mobile AppTesting Challenges
• Highly event-driven, gesture based nature of apps
• Lack of effective automated input gener...
Manual Testing is Still Preferred!!
Automated Mobile Testing Approaches
Record & Replay
AUT/SUT
UI Events
Record & Replay
AUT/SUT
UI Events
Recorder Script
Record & Replay
AUT/SUT
UI Events
Recorder Script
Scripts
Record & Replay
AUT/SUT
UI Events
Recorder Script
Scripts
UI Events
Monkey AUT/SUT
UI Events
Pros and Cons
R&R ✓ Easy reproduction - Expensive collection
and maintenance
- Coupled to locations
Model-BasedTesting
UI Events
Model Monkey AUT/SUT
Model-BasedTesting
UI Events
Model Monkey AUT/SUT
- Manually generated
- Automatically generated (source code)
- Ripped at...
GUI-Rippers
Ripper/Extractor ModelAUT/SUT Monkey
GUI-Rippers
Ripper/Extractor ModelAUT/SUT Monkey
Snapshot 1
GUI-Rippers
Ripper/Extractor ModelAUT/SUT Monkey
Snapshot 1
Snapshot 1
GUI-Rippers
Ripper/Extractor ModelAUT/SUT Monkey
Snapshot 1
Snapshot 1
GUI State 1
GUI-Rippers
Ripper/Extractor ModelAUT/SUT Monkey
Snapshot 1
Snapshot 1
GUI State 1
A or B ?
GUI-Rippers
Ripper/Extractor ModelAUT/SUT Monkey
Snapshot 1
Snapshot 1
GUI State 1
Event 1
A or B ?
GUI-Rippers
Ripper/Extractor ModelAUT/SUT Monkey
Snapshot 1
Snapshot 1
GUI State 1
Event 1
Snapshot 2
Snapshot 2
GUI State...
GUI-Rippers
Monkey
A or B ?
GUI-Rippers
Monkey
A or B ?
Breadth-FirstDepth-First
GUI-Rippers
Monkey
A or B ?
Breadth-FirstDepth-First
Random (Uniform) Random (A-priori distr.)
Other options (online decis...
GUI-Rippers
Ripper/Extractor
Computer/Mobile device
OS
- Framework
- API
- Utilities
GUI State
Events
Monkey
Pros and Cons
R&R ✓Easy reproduction - Expensive collection
and maintenance
- Coupled to locations
MBT ✓ Event sequences
✓...
Pros and Cons
R&R ✓Easy reproduction - Expensive collection
and maintenance
- Coupled to locations
MBT ✓ Event sequences
✓...
Random/FuzzTesting (R/FT)
Monkey
X or Y ?
AUT/SUT
Random/FuzzTesting (R/FT)
Monkey
X or Y ?
AUT/SUT
Event x
Invalid
Random/FuzzTesting (R/FT)
Monkey
X or Y ?
AUT/SUT
Event x
Invalid
Event Y
Valid
Pros and Cons
R&R ✓ Easy reproduction - Expensive collection
and maintenance
- Coupled to locations
MBT ✓ Event sequences
...
Automation Frameworks/APIs (AF/A)
TESTS
JUnit, Espresso, UI Automator, Robotium
Monkey
Automation Frameworks/APIs (AF/A)
UI Automator
Pros and Cons
R&R ✓ Easy reproduction
- Expensive collection
and maintenance
- Coupled to locations
MBT ✓ Event sequences
...
RESEARCH PROTOTYPES
Tool Name Instrumentation GUI Exploration Types of Events
Replayable Test
Cases
Emulators, Devices
Dyn...
OTHER APPROACHES
FOR MOBILE GUI TESTING
CrowdsourcedTesting
AUT/SUTThird Party Service
Remote ManualTesting (Device Farm)
Third Party Service
Google Alpha/BetaTesting
https://support.google.com/googleplay/android-developer/answer/3131213
Tools: Summary
Testing Automation Frameworks/APIs
Crowdsourced Testing/Cloud-based Testing
Record and Replay
Monkey (Fuzz ...
Tools: Summary
Testing Automation Frameworks/APIs
Crowdsourced Testing/Cloud-based Testing
Record and Replay
Monkey (Fuzz ...
Tools: Summary
Testing Automation Frameworks/APIs
Crowdsourced Testing/Cloud-based Testing
Record and Replay
✓ Easy reprod...
Tools: Summary
Testing Automation Frameworks/APIs
Crowdsourced Testing/Cloud-based Testing
Record and Replay
✓ Automation ...
Tools: Summary
Testing Automation Frameworks/APIs
Crowdsourced Testing/Cloud-based Testing
Record and Replay
- Open Source...
Supporting evolution and
maintenance of
Android apps
Mining Android
Software
Repositories
Supporting evolution and
maintenance of
Android apps
Mining Android
Software
Repositories
Android API instability
[FSE’13]...
Supporting evolution and
maintenance of
Android apps
Mining Android
Software
Repositories
Android API instability
[FSE’13]...
SEMERUTools: Monkey Lab
• Testing approach based on natural usages of
an application.
• Models usage using n-gram language...
SEMERUTools: Monkey Lab
• Testing approach based on natural usages of
an application.
• Models usage using n-gram language...
SEMERUTools: FUSION
www.fusion-android.com
• Novel Bug Reporting System for GUI-based
and mobile apps
• Uses a combination...
SEMERUTools: FUSION
www.fusion-android.com
• Novel Bug Reporting System for GUI-based
and mobile apps
• Uses a combination...
SEMERUTools: CRASHSCOPE
• Completely automated testing approach for Android apps
• Generates detailed, expressive bug repo...
SEMERUTools: CRASHSCOPE
• Completely automated testing approach for Android apps
• Generates detailed, expressive bug repo...
SEMERUTools: CRASHSCOPE
• Completely automated testing approach for Android apps
• Generates detailed, expressive bug repo...
Enabling Practical Automated Mobile GUI-Testing
Enabling Practical Automated Mobile GUI-Testing
Essential Challenges in Mobile GUITesting
Essential Challenges
Test Oracles: How does one design an extensible GUI- based
...
Essential Challenges in Mobile GUITesting
Essential Challenges
Test Oracles: How does one design an extensible GUI- based
...
Accidental Challenges in Mobile GUITesting
Accidental Challenges
Application Data and Cold Starts: How can one ensure a st...
Accidental Challenges in Mobile GUITesting
Accidental Challenges
Application Data and Cold Starts: How can one ensure a st...
Essential Challenges:Test Oracles
Potential Solution: Use a hash of the current screen hierarchy
Drawbacks: May miss some ...
Essential Challenges:Test Oracles
Potential Solution: Use a hash of the current screen hierarchy
adb	shell		shell	/system/...
Essential Challenges: Event Coordination
Essential Challenges: Event Coordination
adb	shell	dumpsys	window	-a
mAppTransitionState
APP_STATE_READY
APP_STATE_IDLE
AP...
Essential Challenges: External Features and Dependencies
Potential Solution: Use real usage data from the field?
Accidental Challenges:Application Data and Cold Starts
Accidental Challenges:Application Data and Cold Starts
adb	shell	rm	<app-data-folder>	
adb	shell	pm	clear	<app-package-nam...
Accidental Challenges: Bugs in Framework Utilities
Accidental Challenges: Bugs in Framework Utilities
Accidental Challenges: Bugs in Framework Utilities
adb	shell		shell	/system/bin/uiautomator	
dump	/sdcard/ui_dump.xml
Accidental Challenges: Bugs in Framework Utilities
adb	shell		shell	/system/bin/uiautomator	
dump	/sdcard/ui_dump.xml
Accidental Challenges: Bugs in Framework Utilities
adb	shell		shell	/system/bin/uiautomator	
dump	/sdcard/ui_dump.xml
Solu...
Accidental Challenges: Bugs and Limitations in the SDKTools
adb	Server	1 adb	Server	2
Accidental Challenges: Bugs and Limitations in the SDKTools
adb	Server	1 adb	Server	2
No more than 15 devices per adb serv...
Accidental Challenges: Scaling ConcurrentVirtual Devices
Accidental Challenges: Scaling ConcurrentVirtual Devices
Record>Mine>Generate>Validate Framework
Record>Mine>Generate>Validate Framework
Record>Mine>Generate>Validate Framework
Enabling Large Scale Analysis: Execution Engine
✓Support for multiple
maintenance tasks
✓Open/Closed principle
✓Horizontal...
Enabling Large Scale Analysis: Execution Engine
Dispatcher
Worker
Task
Device/VM/
Thread
token
Task, Device/VM/
Thread tok...
Enabling Large Scale Analysis: Execution Engine
Dispatcher
Worker
Task
Device/VM/
Thread
token
Task, Device/VM/
Thread tok...
THANK YOU!
Kevin Moran
kpmoran@cs.wm.edu
Mario Linares Vasquez
m.linaresv@uniandes.edu.co
Denys Poshyvanyk
denys@cs.wm.edu...
Upcoming SlideShare
Loading in …5
×

Automated GUI-Testing of Android Apps: From Research to Practice

432 views

Published on

The last decade has seen tremendous proliferation of mobile computing in our society. Billions of users have access to millions of mobile applications that can be installed directly on their mobile devices and electrical appliances such as TV set-top boxes. Factors such as new monetization/revenue models, programming models, and distribution infrastructures contribute to an “attractive” movement that captivates new and traditional developers, as well as a crowd of other professionals that explore, design, and implement mobile apps. Also, the need for “enterprise apps” that support start-ups or serve as a new front-end for traditional companies is pushing software-related professionals to embrace the mobile technologies. However, the nature of the economy (devices, apps, markets) imposes new challenges on how mobile apps are envisioned, designed, implemented, tested, released, and maintained. This technology briefing aims to help address the challenges of testing and maintaining mobile apps by providing participants from both academic and industrial backgrounds with information on the state-of-art and state-of-practice mobile testing and maintenance techniques. Specifically, we aim to highlight new techniques and methodologies for making effective automated testing of mobile apps practical and accessible to developers, and the open academic research questions related to such technology transfer.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Automated GUI-Testing of Android Apps: From Research to Practice

  1. 1. ICSME16 Raleigh, NC Wednesday, October 5th, 2016 Kevin Moran, Mario Linares-Vásquez, & Denys Poshyvanyk College of William & Mary - SEMERU - Department of Computer Science Automated GUITesting of Android Apps: From Research to Practice &
  2. 2. GUI-Based Software Testing
  3. 3. UI Events GUITesting: The Main Idea
  4. 4. UI Events Oracle GUITesting: The Main Idea
  5. 5. Output, layout, exceptions, presentation logic, quality attributes, … UI Events Oracle GUITesting: The Main Idea
  6. 6. GUITesting: Core Concepts Test Result Oracle
  7. 7. GUITesting: Core Concepts Test Result Oracle
  8. 8. Monkey Testing !!
  9. 9. Monkey Testing !! Automated Testing !!
  10. 10. Automated GUI-Testing Output, layout, exceptions, presentation logic, quality attributes, … UI Events Monkey
  11. 11. GUI-Based Testing for Mobile Apps
  12. 12. The App Economy http://de.my-walls.net/android-os-robot/
  13. 13. 10
  14. 14. 10
  15. 15. Mobile App Dev Challenges • Rapid platform (and third-party lib) evolution • API-instability and fault-proneness • Influx of unstructured (“low quality”) user reviews • Platform-fragmentation at device and OS level • Quality attributes (performance, energy) and inputs (sensors, camera, WiFi etc.)
  16. 16. Android (andThird Party library) Instability
  17. 17. Study: mining of code changes history (5,848 apps); survey with 45 developers Method: descriptive statistics, statistical hypothesis testing, and open coding Main findings: - Successful apps use more stable and bug free APIs - APIs issues are perceived by users as app programming issues - Lack of tools for identifying API issues and their impact on a new API release Android (andThird Party library) Instability
  18. 18. LargeVolumed of Crowdsourced Requirements and Ratings
  19. 19. Mobile AppTesting Challenges • Highly event-driven, gesture based nature of apps • Lack of effective automated input generation • Lack of robust (semi) automated testing and monitoring tools • Lack of support for debugging apps and bug reporting Joorabchi, M.E.; Mesbah, A.; Kruchten, P., "Real Challenges in Mobile App Development," in Empirical Software Engineering and Measurement, 2013 ACM / IEEE International Symposium on , vol., no., pp.15-24, 10-11 Oct. 2013
  20. 20. Manual Testing is Still Preferred!!
  21. 21. Automated Mobile Testing Approaches
  22. 22. Record & Replay AUT/SUT UI Events
  23. 23. Record & Replay AUT/SUT UI Events Recorder Script
  24. 24. Record & Replay AUT/SUT UI Events Recorder Script Scripts
  25. 25. Record & Replay AUT/SUT UI Events Recorder Script Scripts UI Events Monkey AUT/SUT UI Events
  26. 26. Pros and Cons R&R ✓ Easy reproduction - Expensive collection and maintenance - Coupled to locations
  27. 27. Model-BasedTesting UI Events Model Monkey AUT/SUT
  28. 28. Model-BasedTesting UI Events Model Monkey AUT/SUT - Manually generated - Automatically generated (source code) - Ripped at runtime (upfront) - Ripped at runtime (interactive)
  29. 29. GUI-Rippers Ripper/Extractor ModelAUT/SUT Monkey
  30. 30. GUI-Rippers Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1
  31. 31. GUI-Rippers Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1 Snapshot 1
  32. 32. GUI-Rippers Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1 Snapshot 1 GUI State 1
  33. 33. GUI-Rippers Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1 Snapshot 1 GUI State 1 A or B ?
  34. 34. GUI-Rippers Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1 Snapshot 1 GUI State 1 Event 1 A or B ?
  35. 35. GUI-Rippers Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1 Snapshot 1 GUI State 1 Event 1 Snapshot 2 Snapshot 2 GUI State 2 A or B ?
  36. 36. GUI-Rippers Monkey A or B ?
  37. 37. GUI-Rippers Monkey A or B ? Breadth-FirstDepth-First
  38. 38. GUI-Rippers Monkey A or B ? Breadth-FirstDepth-First Random (Uniform) Random (A-priori distr.) Other options (online decision)
  39. 39. GUI-Rippers Ripper/Extractor Computer/Mobile device OS - Framework - API - Utilities GUI State Events Monkey
  40. 40. Pros and Cons R&R ✓Easy reproduction - Expensive collection and maintenance - Coupled to locations MBT ✓ Event sequences ✓ Automatic exploration - Invalid sequences (a- priori model) - Few corner cases - Use cases are not considered - Manual execution - Incomplete models
  41. 41. Pros and Cons R&R ✓Easy reproduction - Expensive collection and maintenance - Coupled to locations MBT ✓ Event sequences ✓ Automatic exploration - Invalid sequences (a- priori model) - Few corner cases - Use cases are not considered - Manual execution - Incomplete models Rippers - Lack of context - No history awareness
  42. 42. Random/FuzzTesting (R/FT) Monkey X or Y ? AUT/SUT
  43. 43. Random/FuzzTesting (R/FT) Monkey X or Y ? AUT/SUT Event x Invalid
  44. 44. Random/FuzzTesting (R/FT) Monkey X or Y ? AUT/SUT Event x Invalid Event Y Valid
  45. 45. Pros and Cons R&R ✓ Easy reproduction - Expensive collection and maintenance - Coupled to locations MBT ✓ Event sequences ✓ Automatic exploration - Invalid sequences (a- priori model) - Few corner cases - Use cases are not considered - Manual execution - Incomplete models Rippers - Lack of context - No history awareness R/FT ✓ Fast execution ✓ Good at finding crashes - Invalid events - Lack of expressiveness
  46. 46. Automation Frameworks/APIs (AF/A) TESTS JUnit, Espresso, UI Automator, Robotium Monkey
  47. 47. Automation Frameworks/APIs (AF/A) UI Automator
  48. 48. Pros and Cons R&R ✓ Easy reproduction - Expensive collection and maintenance - Coupled to locations MBT ✓ Event sequences ✓ Automatic exploration - Invalid sequences (a-priori model) - Few corner cases - Use cases are not considered - Manual execution - Incomplete models Rippers - Lack of context - No history awareness R/FT ✓ Fast execution ✓ Good at finding crashes - Invalid events - Lack of expressiveness AF/A ✓ Easy reproduction ✓ High level syntax ✓ Black box testing - Learning curve - User-defined oracles - Expensive maintenance
  49. 49. RESEARCH PROTOTYPES Tool Name Instrumentation GUI Exploration Types of Events Replayable Test Cases Emulators, Devices Dynodroid Yes Guided/Random System, GUI, Text No No EvoDroid No System/Evo GUI No N/A AndroidRipper Yes Systematic GUI, Text No N/A MobiGUItar Yes Model-Based GUI, Text Yes N/A A3E Depth-First Yes Systematic GUI No Yes A3E Targeted Yes Model-Based GUI No Yes Swifthand Yes Model-Based GUI, Text No Yes PUMA Yes Programmable System, GUI, Text No Yes ACTEve Yes Systematic GUI No Yes VANARSena Yes Random System, GUI, Text Yes N/A Thor Yes Test Cases Test Case Events N/A No QUANTUM Yes Model-Based System, GUI Yes N/A AppDoctor Yes Multiple System, GUI2, Text Yes N/A ORBIT No Model-Based GUI No N/A SPAG-C No Record/Replay GUI N/A No JPF-Android No Scripting GUI Yes N/A MonkeyLab No Model-based GUI, Text Yes Yes CrashDroid No Manual Rec/Replay GUI, Text Yes Yes SIG-Droid No Symbolic GUI, Text Yes N/A CrashScope No Systematic GUI, Text, System Yes Yes Sapienz Yes Evolutionary GUI, Text,System Yes Yes
  50. 50. OTHER APPROACHES FOR MOBILE GUI TESTING
  51. 51. CrowdsourcedTesting AUT/SUTThird Party Service
  52. 52. Remote ManualTesting (Device Farm) Third Party Service
  53. 53. Google Alpha/BetaTesting https://support.google.com/googleplay/android-developer/answer/3131213
  54. 54. Tools: Summary Testing Automation Frameworks/APIs Crowdsourced Testing/Cloud-based Testing Record and Replay Monkey (Fuzz testing)
  55. 55. Tools: Summary Testing Automation Frameworks/APIs Crowdsourced Testing/Cloud-based Testing Record and Replay Monkey (Fuzz testing) ✓ Unexpected scenarios - Reproducibility/Expressiveness
  56. 56. Tools: Summary Testing Automation Frameworks/APIs Crowdsourced Testing/Cloud-based Testing Record and Replay ✓ Easy reproduction ✓ Automation on different devices - Test scripts maintenance - Learning curve Monkey (Fuzz testing) ✓ Unexpected scenarios - Reproducibility/Expressiveness
  57. 57. Tools: Summary Testing Automation Frameworks/APIs Crowdsourced Testing/Cloud-based Testing Record and Replay ✓ Automation on different devices ✓ Similar to manual testing - Test scripts maintenance ✓ Easy reproduction ✓ Automation on different devices - Test scripts maintenance - Learning curve Monkey (Fuzz testing) ✓ Unexpected scenarios - Reproducibility/Expressiveness
  58. 58. Tools: Summary Testing Automation Frameworks/APIs Crowdsourced Testing/Cloud-based Testing Record and Replay - Open Source vs Paid ✓ Large set of devices ✓ Remote data collection ✓ Support automation frameworks ✓ Automation on different devices ✓ Similar to manual testing - Test scripts maintenance ✓ Easy reproduction ✓ Automation on different devices - Test scripts maintenance - Learning curve Monkey (Fuzz testing) ✓ Unexpected scenarios - Reproducibility/Expressiveness
  59. 59. Supporting evolution and maintenance of Android apps Mining Android Software Repositories
  60. 60. Supporting evolution and maintenance of Android apps Mining Android Software Repositories Android API instability [FSE’13][TSE’15] Performance bottlenecks [ICSME’15] [MSR’16] User reviews [ICSME’15] Energy [MSR’14]
  61. 61. Supporting evolution and maintenance of Android apps Mining Android Software Repositories Android API instability [FSE’13][TSE’15] Performance bottlenecks [ICSME’15] [MSR’16] User reviews [ICSME’15] Energy [MSR’14] Crowdsource-based requirements [ICSME’15] GUI - based testing [MSR’15] Bug reporting [FSE’15] Optimizing energy consumption [FSE’15]
  62. 62. SEMERUTools: Monkey Lab • Testing approach based on natural usages of an application. • Models usage using n-gram language models to produce unseen yet natural event sequences. • Capable of generating orthogonal test cases to those already modeled.
  63. 63. SEMERUTools: Monkey Lab • Testing approach based on natural usages of an application. • Models usage using n-gram language models to produce unseen yet natural event sequences. • Capable of generating orthogonal test cases to those already modeled.
  64. 64. SEMERUTools: FUSION www.fusion-android.com • Novel Bug Reporting System for GUI-based and mobile apps • Uses a combination of static and dynamic analysis to help inform the reporting process • Uses auto-completion to ease the burden on reporters and ensure high-quality reports.
  65. 65. SEMERUTools: FUSION www.fusion-android.com • Novel Bug Reporting System for GUI-based and mobile apps • Uses a combination of static and dynamic analysis to help inform the reporting process • Uses auto-completion to ease the burden on reporters and ensure high-quality reports.
  66. 66. SEMERUTools: CRASHSCOPE • Completely automated testing approach for Android apps • Generates detailed, expressive bug reports and repayable scripts • A practical tool, requiring no instrumentation framework, or modification to the OS or applications • Capable of running on both physical devices and emulators • Differing execution strategies able to test contextual features
  67. 67. SEMERUTools: CRASHSCOPE • Completely automated testing approach for Android apps • Generates detailed, expressive bug reports and repayable scripts • A practical tool, requiring no instrumentation framework, or modification to the OS or applications • Capable of running on both physical devices and emulators • Differing execution strategies able to test contextual features
  68. 68. SEMERUTools: CRASHSCOPE • Completely automated testing approach for Android apps • Generates detailed, expressive bug reports and repayable scripts • A practical tool, requiring no instrumentation framework, or modification to the OS or applications • Capable of running on both physical devices and emulators • Differing execution strategies able to test contextual features
  69. 69. Enabling Practical Automated Mobile GUI-Testing
  70. 70. Enabling Practical Automated Mobile GUI-Testing
  71. 71. Essential Challenges in Mobile GUITesting Essential Challenges Test Oracles: How does one design an extensible GUI- based oracle that can be used to verify expected application behavior across different device configurations? Event Coordination: In automated GUI-based testing, deciding the proper timing of firing events on a device is challenging. External Features: How can external features such as notifications be properly incorporated into large-scale testing of mobile apps? External Dependencies: Many mobile apps use external servers or databases in order to send, store, and receive information. How can these dependencies be appropriately incorporated into large-scale testing?
  72. 72. Essential Challenges in Mobile GUITesting Essential Challenges Test Oracles: How does one design an extensible GUI- based oracle that can be used to verify expected application behavior across different device configurations? Event Coordination: In automated GUI-based testing, deciding the proper timing of firing events on a device is challenging. External Features: How can external features such as notifications be properly incorporated into large-scale testing of mobile apps? External Dependencies: Many mobile apps use external servers or databases in order to send, store, and receive information. How can these dependencies be appropriately incorporated into large-scale testing?
  73. 73. Accidental Challenges in Mobile GUITesting Accidental Challenges Application Data and Cold Starts: How can one ensure a sterile or desired testing environment within the context of large scale testing? Bugs in Framework Utilities: There exist certain limitations with core Android Frameworks popular with GUI-testing. Bugs and Limitations in the SDK Tools: There are limitations inherent to Android developer tools, such as adb, that make large-scale parallel execution and testing difficult. Challenges Scaling Concurrent Virtual Devices: Android emulators were not designed to scale well on modest hardware.
  74. 74. Accidental Challenges in Mobile GUITesting Accidental Challenges Application Data and Cold Starts: How can one ensure a sterile or desired testing environment within the context of large scale testing? Bugs in Framework Utilities: There exist certain limitations with core Android Frameworks popular with GUI-testing. Bugs and Limitations in the SDK Tools: There are limitations inherent to Android developer tools, such as adb, that make large-scale parallel execution and testing difficult. Challenges Scaling Concurrent Virtual Devices: Android emulators were not designed to scale well on modest hardware.
  75. 75. Essential Challenges:Test Oracles Potential Solution: Use a hash of the current screen hierarchy Drawbacks: May miss some fined grained information about the GUI (e.g. colors, images)
  76. 76. Essential Challenges:Test Oracles Potential Solution: Use a hash of the current screen hierarchy adb shell shell /system/bin/uiautomator dump /sdcard/ui_dump.xml Drawbacks: May miss some fined grained information about the GUI (e.g. colors, images)
  77. 77. Essential Challenges: Event Coordination
  78. 78. Essential Challenges: Event Coordination adb shell dumpsys window -a mAppTransitionState APP_STATE_READY APP_STATE_IDLE APP_STATE_TIMEOUT APP_STATE_RUNNING
  79. 79. Essential Challenges: External Features and Dependencies Potential Solution: Use real usage data from the field?
  80. 80. Accidental Challenges:Application Data and Cold Starts
  81. 81. Accidental Challenges:Application Data and Cold Starts adb shell rm <app-data-folder> adb shell pm clear <app-package-name>
  82. 82. Accidental Challenges: Bugs in Framework Utilities
  83. 83. Accidental Challenges: Bugs in Framework Utilities
  84. 84. Accidental Challenges: Bugs in Framework Utilities adb shell shell /system/bin/uiautomator dump /sdcard/ui_dump.xml
  85. 85. Accidental Challenges: Bugs in Framework Utilities adb shell shell /system/bin/uiautomator dump /sdcard/ui_dump.xml
  86. 86. Accidental Challenges: Bugs in Framework Utilities adb shell shell /system/bin/uiautomator dump /sdcard/ui_dump.xml Solution: Modify Framework Components for the AOSP
  87. 87. Accidental Challenges: Bugs and Limitations in the SDKTools adb Server 1 adb Server 2
  88. 88. Accidental Challenges: Bugs and Limitations in the SDKTools adb Server 1 adb Server 2 No more than 15 devices per adb server instance!
  89. 89. Accidental Challenges: Scaling ConcurrentVirtual Devices
  90. 90. Accidental Challenges: Scaling ConcurrentVirtual Devices
  91. 91. Record>Mine>Generate>Validate Framework
  92. 92. Record>Mine>Generate>Validate Framework
  93. 93. Record>Mine>Generate>Validate Framework
  94. 94. Enabling Large Scale Analysis: Execution Engine ✓Support for multiple maintenance tasks ✓Open/Closed principle ✓Horizontal/Vertical scalability ✓Asynchronous communication between clients and execution engine ✓Support for fragmentation Message bus Execution Engine Virtual devices (Pool)
  95. 95. Enabling Large Scale Analysis: Execution Engine Dispatcher Worker Task Device/VM/ Thread token Task, Device/VM/ Thread token Task status, response Task status - Task-specific frameworks - Third party libraries <<invokes>> Client i Message Bus Local DB Client-specific technologies Request (i.e., new Task) Web folder or Content Delivery Network Images, files, etc.Images, files, etc. Response (i.e., Task results and status) Execution Engine (EE) EE Settings Devices/VMs pool Background Threads pool Clients Client n
  96. 96. Enabling Large Scale Analysis: Execution Engine Dispatcher Worker Task Device/VM/ Thread token Task, Device/VM/ Thread token Task status, response Task status - Task-specific frameworks - Third party libraries <<invokes>> Client i Message Bus Local DB Client-specific technologies Request (i.e., new Task) Web folder or Content Delivery Network Images, files, etc.Images, files, etc. Response (i.e., Task results and status) Execution Engine (EE) EE Settings Devices/VMs pool Background Threads pool Clients Client n
  97. 97. THANK YOU! Kevin Moran kpmoran@cs.wm.edu Mario Linares Vasquez m.linaresv@uniandes.edu.co Denys Poshyvanyk denys@cs.wm.edu http://www.research-appendix.com/android-testing

×