Testing for Android: When, Where, and How to Successfully Use Test Automation

829 views

Published on

A high-level overview of test automation on the Android platform, beginning with a brief intro to software testing, how mobile isn't all that different from problems faced in the past, a tour of the most popular free tools available, and some words of advice.

The slides accompanied a talk presented at the San Francisco Android User Group on October 29, hosted at Twitter.

Published in: Technology
  • Be the first to comment

Testing for Android: When, Where, and How to Successfully Use Test Automation

  1. 1. Testing for Android When, Where, and How to Successfully Use Test Automation Trent Peterson | @tdpeterson | trent@appthwack.com
  2. 2. Who?
  3. 3. AGENDA What is testing? What’s the problem? Why should I care? When: Choosing our battles Where: Best Practices How: Tools & Frameworks Q&A
  4. 4. INTRO TO TESTING
  5. 5. Types of tests • • • • • • • • Functional Non-functional Integration Unit Regression Smoke Usability Performance • Stress • Acceptance • Load • Sanity • End-to-End • System • Security And on and on…
  6. 6. Testing distilled for this discussion Functional Testing Non-functional Testing
  7. 7. THE PROBLEM
  8. 8. Building things is hard.
  9. 9. WHY SHOULD I CARE?
  10. 10. So what’s the big deal? 1,000,000+ apps in 2 crashes and 84% will uninstall - Compuware as reported in TechCrunch
  11. 11. We’ve been here before.
  12. 12. CHOOSING OUR BATTLES
  13. 13. First, what’s our goal? Save time! BE REALISTIC Save money! • Increase regularity of testing Shrink QA! • Free up resources for other testing efforts • Enable previously impossible tests AUTOMATE EVERYTHING !
  14. 14. Deciding what to automate How often will I test this? Is this something a human is good at? How much effort will automation require?
  15. 15. Some examples AUTOMATE MANUAL • Granular functionality (unit tests) • Repetitive tasks (update paths, navigation, etc.) • Performance • UX • Responsiveness and “feel” • New functionality
  16. 16. Our options (Machine) JVM Emulators Real Devices • FAST • Unit tests only* • Mocks • Cheap • Catch layout issues • Simulate calls/SMS • SLOW • Illusion of 1:1 with real devices • Exactly what end customers use • Performance data • Fast • Expensive: Purchase, maintain, etc.
  17. 17. Our options (Human) QA Crowd End Users • Pros • Real feedback • Know the product • Expensive • Real feedback • Might know product • Expensive (usually) • Slow • Don’t do this.
  18. 18. BEST PRACTICES
  19. 19. No barriers between Dev & QA
  20. 20. Automation is a software project
  21. 21. Automation is a tool, not a solution
  22. 22. Automate with precision
  23. 23. It’s all about data
  24. 24. Know your matrix
  25. 25. Abstraction is important …but don’t obsess
  26. 26. TOOLS and FRAMEWORKS
  27. 27. Tool: Recorders GOOD BAD • Fast • Non-developers can create tests • Fragile • Difficult to maintain • Non-developers can create tests BEST PRACTICE • Treat as a fancy spy
  28. 28. Tool: UI Exerciser Monkey $ adb shell monkey –p my.sweet.app –v 1000
  29. 29. Framework: Android Testing Framework SUMMARY • JUnit/Java • Instrumentation provides hooks into Android SDK to control Android-specific functionality • Very basic functionality supported • The foundation for most Android testing. Learn it, use it, love it.
  30. 30. Framework: UI Automator SUMMARY • JUnit/Java • Extends testing framework with UI selectors and manipulators • Handle/control system dialogs, etc • 4.1+ only • Excellent choice if you have the luxury of supporting 4.1+.
  31. 31. Framework: Robolectric SUMMARY • JUnit • Runs in JVM • Mocks Android functionality with “shadow” objects • Super fast • Not everything is mocked (also, mocks) • Great for staying sane while developing
  32. 32. Framework: Robotium SUMMARY • JUnit extension • Runs on emulators and real devices • Oriented towards black-box testing and ensuring real-world outcomes • Limited to the app under test • Great choice for developers going beyond basic unit testing and testing UI.
  33. 33. Framework: Cucumber-based SUMMARY • Human-readable tests • Runs on emulators and real devices • Cross-platform • Complex scenarios require development • Behavior-driven design. Be careful! • Great choice for simple flows and teams with limited developer resources.
  34. 34. Framework: Appium SUMMARY • Supports many language adapters • Runs on emulators and real devices • Cross-platform • New, and not all WebDriver concepts map intuitively to native apps • Great choice for teams happy with Selenium and expanding to native.
  35. 35. Frameworks: Commercial GOOD BAD • Support • Added features and integrations • Longevity? • Proprietary • Often heavy and difficult to maintain • $$$ ADVICE • Evaluate very carefully and acknowledge lock-in.
  36. 36. Tool: Spoon Spoon Instrumentation Tests
  37. 37. When choosing, ask… What are we testing? What are our automation goals? Who will write the automated tests?
  38. 38. SHAMELESS PLUG
  39. 39. API • Continuous Integration • IDE • Command Line Clients • Browser-based web app AppThwack • • • • 100s of non-rooted devices Parallel execution Built-in compatibility tests Support for all popular automation frameworks Results • Interactive, realtime report • JSON data via API
  40. 40. Q&A Trent Peterson | @tdpeterson | trent@appthwack.com

×