Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ICSE17 - Tool Demonstration - CrashScope A Practical Tool for the Automated Testing of Android Applications

156 views

Published on

This presentation covers the CrashScope tool which uses a combination of static and dynamic analysis to practically and effectively test Android applications. For more information please visit https://www.android-dev-tools.com/crashscope-home/

Published in: Software
  • Be the first to comment

ICSE17 - Tool Demonstration - CrashScope A Practical Tool for the Automated Testing of Android Applications

  1. 1. ICSE’17 Buenos Aires,Argentina Wednesday, May 24th, 2017 Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, ChristopherVendome, & Denys Poshyvanyk CrashScope:A PracticalTool for AutomatedTesting of Android Applications
  2. 2. 3
  3. 3. 3
  4. 4. MANUALTESTING
  5. 5. MANUALTESTING
  6. 6. AUTOMATEDTESTING
  7. 7. AUTOMATEDTESTING
  8. 8. AUTOMATEDTESTING
  9. 9. CATEGORIES OF AUTOMATEDTESTING APPROACHES FOR MOBILE APPS • Random-based • Systematic-based • Model-based • Record and replay • Others (ManualTesting Frameworks)
  10. 10. THE CURRENT STATE OF AUTOMATED MOBILE APPLICATIONTESTING Tool Name Instr. GUI Exploration Types of Events Crash Resilient Replayable Test Cases NL Crash Reports Emulators, Devices Dynodroid Yes Guided/Random System, GUI, Text Yes No No No EvoDroid No System/Evo GUI No No No N/A AndroidRipper Yes Systematic GUI, Text No No No N/A MobiGUItar Yes Model-Based GUI, Text No Yes No N/A A3E DFS Yes Systematic GUI No No No Yes A3E Targeted [20] Yes Model-Based GUI No No No Yes Swifthand Yes Model-Based GUI, Text N/A No No Yes PUMA Yes Programmable System, GUI, Text N/A No No Yes ACTEve Yes Systematic GUI N/A No No Yes VANARSena Yes Random System, GUI, Text Yes Yes No N/A Thor Yes Test Cases Test Case Events N/A N/A No No QUANTUM Yes Model-Based System, GUI N/A Yes No N/A AppDoctor Yes Multiple System, GUI, Text Yes Yes No N/A ORBIT No Model-Based GUI N/A No No N/A SPAG-C No Record/Replay GUI N/A N/A No No JPF-Android No Scripting GUI N/A Yes No N/A MonkeyLab No Model-based GUI, Text No Yes No Yes CrashDroid No Manual Rec/Replay GUI, Text Manual Yes Yes Yes SIG-Droid No Symbolic GUI, Text N/A Yes No N/A Sapienz No Search-Based GUI, Text N/A Yes No Yes CrashScope No Systematic GUI, Text, System Yes Yes Yes Yes
  11. 11. THE CURRENT STATE OF AUTOMATED MOBILE APPLICATIONTESTING Tool Name Instr. GUI Exploration Types of Events Crash Resilient Replayable Test Cases NL Crash Reports Emulators, Devices Dynodroid Yes Guided/Random System, GUI, Text Yes No No No EvoDroid No System/Evo GUI No No No N/A AndroidRipper Yes Systematic GUI, Text No No No N/A MobiGUItar Yes Model-Based GUI, Text No Yes No N/A A3E DFS Yes Systematic GUI No No No Yes A3E Targeted [20] Yes Model-Based GUI No No No Yes Swifthand Yes Model-Based GUI, Text N/A No No Yes PUMA Yes Programmable System, GUI, Text N/A No No Yes ACTEve Yes Systematic GUI N/A No No Yes VANARSena Yes Random System, GUI, Text Yes Yes No N/A Thor Yes Test Cases Test Case Events N/A N/A No No QUANTUM Yes Model-Based System, GUI N/A Yes No N/A AppDoctor Yes Multiple System, GUI, Text Yes Yes No N/A ORBIT No Model-Based GUI N/A No No N/A SPAG-C No Record/Replay GUI N/A N/A No No JPF-Android No Scripting GUI N/A Yes No N/A MonkeyLab No Model-based GUI, Text No Yes No Yes CrashDroid No Manual Rec/Replay GUI, Text Manual Yes Yes Yes SIG-Droid No Symbolic GUI, Text N/A Yes No N/A Sapienz No Search-Based GUI, Text N/A Yes No Yes CrashScope No Systematic GUI, Text, System Yes Yes Yes Yes What are the limitations of current automated approaches?
  12. 12. LIMITATIONS OF AUTOMATED MOBILE TESTING AND DEBUGGING 1S. R. Choudhary,A. Gorla, and A. Orso.AutomatedTest Input Generation for Android:Are we there yet? In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015), 2015
  13. 13. LIMITATIONS OF AUTOMATED MOBILE TESTING AND DEBUGGING • Lack of detailed, easy to understand testing results for faults/ crashes1 1S. R. Choudhary,A. Gorla, and A. Orso.AutomatedTest Input Generation for Android:Are we there yet? In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015), 2015
  14. 14. LIMITATIONS OF AUTOMATED MOBILE TESTING AND DEBUGGING • Lack of detailed, easy to understand testing results for faults/ crashes1 • No easy way to reproduce test scenarios1 1S. R. Choudhary,A. Gorla, and A. Orso.AutomatedTest Input Generation for Android:Are we there yet? In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015), 2015
  15. 15. LIMITATIONS OF AUTOMATED MOBILE TESTING AND DEBUGGING • Lack of detailed, easy to understand testing results for faults/ crashes1 • No easy way to reproduce test scenarios1 • Not practical from a developers viewpoint 1S. R. Choudhary,A. Gorla, and A. Orso.AutomatedTest Input Generation for Android:Are we there yet? In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015), 2015
  16. 16. LIMITATIONS OF AUTOMATED MOBILE TESTING AND DEBUGGING • Lack of detailed, easy to understand testing results for faults/ crashes1 • No easy way to reproduce test scenarios1 • Not practical from a developers viewpoint • Few approaches enable different strategies capable of generating text and testing contextual features 1S. R. Choudhary,A. Gorla, and A. Orso.AutomatedTest Input Generation for Android:Are we there yet? In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015), 2015
  17. 17. PAST STUDIES OF MOBILE CRASHES AND BUGS • Many crashes can be mapped to well-defined, externally inducible faults1 • Contextual features, such as network connectivity and screen rotation, account for many of these externally inducible faults12 • These dominant root causes can affect many different user execution paths1 1L. Ravindranath, S. Nath, J. Padhye, and H. Balakrishnan.Automatic and scalable fault detection for mobile applications. MobiSys ’14 2R. N. Zaeem, M. R. Prasad, and S. Khurshid.Automated generation of oracles for testing user-interaction features of mobile apps, ICST ’14
  18. 18. OUR SOLUTION: CRASHSCOPE
  19. 19. OUR SOLUTION: CRASHSCOPE • Completely automated approach
  20. 20. OUR SOLUTION: CRASHSCOPE • Completely automated approach • Generates detailed, expressive bug reports and repayable scripts
  21. 21. OUR SOLUTION: CRASHSCOPE • Completely automated approach • Generates detailed, expressive bug reports and repayable scripts • A practical tool, requiring no instrumentation framework, or modification to the OS or applications
  22. 22. OUR SOLUTION: CRASHSCOPE • Completely automated approach • Generates detailed, expressive bug reports and repayable scripts • A practical tool, requiring no instrumentation framework, or modification to the OS or applications • Capable of running on both physical devices and emulators
  23. 23. OUR SOLUTION: CRASHSCOPE • Completely automated approach • Generates detailed, expressive bug reports and repayable scripts • A practical tool, requiring no instrumentation framework, or modification to the OS or applications • Capable of running on both physical devices and emulators • Differing execution strategies able to test contextual features
  24. 24. CRASHSCOPE DESIGN CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine Physical Device or Emulator .apk app src or Android Application Report Generation Crash-Execution Script Generator Crash-Execution Script Replayer 1 2 3 4 5
  25. 25. CRASHSCOPE DESIGN CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine Physical Device or Emulator .apk app src or Android Application Report Generation Crash-Execution Script Generator Crash-Execution Script Replayer 1 2 3 4 5 1
  26. 26. CRASHSCOPE DESIGN CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine Physical Device or Emulator .apk app src or Android Application Report Generation Crash-Execution Script Generator Crash-Execution Script Replayer 1 2 3 4 5 1 II
  27. 27. CRASHSCOPE:ANALYSIS GUI Ripping Engine .apk or app src Physical Device or Emulator Android UIAutomator Event Execution Engine (adb input & telnet) —Touch Event —GUI Component Information —Screenshots Crash after last step? YesNo Execution Finished? No Yes Decision Engine Determine next <Action, GUI> Event to Execute Enable/Disable Activity/App Features Save Execution Information 2 Continue Execution CrashScope Database 3 D Contextual Feature Extractor1 .apk decompiler (if necessary) Android Application Manifest File Parser API Extractor Rotatable Activities App and Activity Level Contextual Features App and Activity Level Contextual Features
  28. 28. CRASHSCOPE: EXPLORATION
  29. 29. CRASHSCOPE: EXPLORATION
  30. 30. CRASHSCOPE: EXPLORATION
  31. 31. CRASHSCOPE: EXPLORATION uiautomator
  32. 32. CRASHSCOPE: EXPLORATION uiautomator
  33. 33. CRASHSCOPE: EXPLORATION
  34. 34. CRASHSCOPE: EXPLORATION
  35. 35. • Activity • Checkable, Checked, Clickable, Long Clickable? • Component Index • Current Window • Enabled? • XML_ID • ComponentType • Position (Absolute and Relative) • Text • Screenshot → CRASHSCOPE: EXPLORATION
  36. 36. CRASHSCOPE: EXPLORATION
  37. 37. CRASHSCOPE: EXPLORATION CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine .apk app src or Android Application 1 2 3 4
  38. 38. CRASHSCOPE STRATEGIES • GUI-Traversal:Top-Down & Bottom Up (DFS) • Text Entry: Expected, Unexpected, NoText • Contextual Features: Enabled or Disabled
  39. 39. CRASHSCOPE DEMO
  40. 40. CRASHSCOPE DEMO
  41. 41. CRASHSCOPE: REPORT AND SCRIPT GENERATION Augmented Natural Language Report Generator Crash Execution Script Generator Web Based Application Bug Report (JSP, MySQL, and Bootstrap) Crash Execution Script Replayer Googlehttp://cs.wm.edu/semeru CrashScope Report Database Parser CrashScope Script Generator Replay Script Parser Contextual Event Interperter / adb Replayer Physical Device or Emulator Contextual Event Execution (telnet commands) Event Execution Engine (adb sendevent & adb input) 4 5 6 7 CrashScope Database 3 Step Processor Database Parser App Executions Containing Crashes Replay Script Tuples <adb shell input tap 780 1126> <adb shell input text ‘abc!@#’> <Disable_Network> <Disable_GPS> App Executions Containing Crashes
  42. 42. CRASHSCOPE: REPORTS
  43. 43. CRASHSCOPE: REPORTS
  44. 44. CRASHSCOPE: REPORTS
  45. 45. WEB-APP ARCHITECTURE CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine Pool of Physical Devices or Virtual Android Devices .apk app srcor Upload Android App & Specify Strategies Report Generation Crash Execution Script Generator Crash-Execution Script Replayer 1 Web Application Back-End (Execution Engine) Detailed Crash Report 2 4 5 3 6 7 Java Web App Java Web App CouchDB (Message Bus) PollingforTasks Json ExecutionResults Json Json Json adb script
  46. 46. WEB-APP ARCHITECTURE CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine Pool of Physical Devices or Virtual Android Devices .apk app srcor Upload Android App & Specify Strategies Report Generation Crash Execution Script Generator Crash-Execution Script Replayer 1 Web Application Back-End (Execution Engine) Detailed Crash Report 2 4 5 3 6 7 Java Web App Java Web App CouchDB (Message Bus) PollingforTasks Json ExecutionResults Json Json Json adb script Java Web App
  47. 47. WEB-APP ARCHITECTURE CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine Pool of Physical Devices or Virtual Android Devices .apk app srcor Upload Android App & Specify Strategies Report Generation Crash Execution Script Generator Crash-Execution Script Replayer 1 Web Application Back-End (Execution Engine) Detailed Crash Report 2 4 5 3 6 7 Java Web App Java Web App CouchDB (Message Bus) PollingforTasks Json ExecutionResults Json Json Json adb script Java Web App CrashScope Back-end
  48. 48. CRASHSCOPE:A PRACTICALTOOL
  49. 49. CRASHSCOPE:A PRACTICALTOOL
  50. 50. EVALUATION • Two Empirical Studies • Study1: Crash Detection Capabilities • Study 2: Crash Report Reproducibility and Readability
  51. 51. STUDY 1: CRASH DETECTION & COVERAGE • RQ1: Crash Detection Effectiveness?
 • RQ2: Orthogonality of Crashes?
 • RQ3: Effectiveness of Individual Strategies?
 • RQ4: Does Crash Detection Correlate with Code Coverage?
  52. 52. STUDY 1: EXPERIMENTAL SETUP • 61 subject applications from the Androtest1 toolset • Each testing tool was run 5 separate times for 1 hour, whereas CrashScope ran through all strategies • Monkey was limited by the number of events Tool Name Android Version Tool Type Monkey Any Random A3E Depth-First Any Systematic GUI-Ripper Any Model-Based Dynodroid v2.3 Random-Based PUMA v4.1+ Random-Based TOOLS USED IN THE COMPARATIVE FAULT FINDING STUDY 1S. R. Choudhary,A. Gorla, and A. Orso.AutomatedTest Input Generation for Android:Are we there yet? In 30th IEEE/ACM International Conference on Automated Software Engineering (ASE 2015), 2015
  53. 53. STUDY 1: SUMMARY OF FINDINGS • RQ1: CrashScope is nearly as effective at discovering crashes as the other tools, without reporting crashes caused by instrumentation • RQ2&3: CrashScope’s differing strategies led to the discovery of unique crashes • RQ4: Higher statement coverage does not necessarily correspond with crash detection capabilities
  54. 54. STUDY 2: REPRODUCIBILITY & READABILITY • RQ5: Reproducibility of CrashScope Reports?
 
 • RQ6:Readability of CrashScope Reports?

  55. 55. STUDY 2: SUMMARY OF FINDINGS • RQ5: Reports generated by CrashScope are about as reproducible as human written reports extracted from open-source issue trackers
 
 • RQ6:Reports generated by CrashScope are more readable and useful from a developers’ perspective compared to human-written reports.

  56. 56. DETAILED FINDINGS ICST’16 Technical Paper
  57. 57. CONCLUSION CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine Physical Device or Emulator .apk app src or Android Application Report Generation Crash-Execution Script Generator Crash-Execution Script Replayer 1 2 3 4 5 Physical Device or Emulator
  58. 58. CONCLUSION CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine Physical Device or Emulator .apk app src or Android Application Report Generation Crash-Execution Script Generator Crash-Execution Script Replayer 1 2 3 4 5 Physical Device or Emulator
  59. 59. CONCLUSION CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine Physical Device or Emulator .apk app src or Android Application Report Generation Crash-Execution Script Generator Crash-Execution Script Replayer 1 2 3 4 5 Physical Device or Emulator
  60. 60. CONCLUSION CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine Physical Device or Emulator .apk app src or Android Application Report Generation Crash-Execution Script Generator Crash-Execution Script Replayer 1 2 3 4 5 Physical Device or Emulator
  61. 61. CONCLUSION CrashScope Database Static Analysis (Contextual Feature Extraction) GUI-Ripping Engine Physical Device or Emulator .apk app src or Android Application Report Generation Crash-Execution Script Generator Crash-Execution Script Replayer 1 2 3 4 5 Physical Device or Emulator
  62. 62. Any Questions? Thank you! We are working to Open Source CrashScope! please visit: Kevin Moran Ph.D. candidate kpmoran@cs.wm.edu http://www.cs.wm.edu/~kpmoran https://www.android-dev-tools.com/crashscope-home/

×