When & How to Successfully use Test Automation for Mobile Applications

1,249
-1

Published on

A free webinar produced by the Technology Association of Oregon's Quality Assurance Forum Group. Presented by Trent Peterson, CEO and Co-Founder of Portland-based AppThwack, providing fast automated mobile device testing services for Android, iPhone, iPad, and mobile web apps. The recording to the webinar can be accessed here: http://youtu.be/r9Vm6ssSEU8

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,249
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
51
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Picture courtesy of http://www.flickr.com/photos/schrierc/ (Creative Commons Attribution 2.0 Generic)
  • Picture courtesy of http://www.flickr.com/photos/conbon/ (Creative Commons Attribution 2.0 Generic)
  • Programming image by http://www.flickr.com/photos/riebart/ (Creative Commons Attribution 2.0 Generic)
  • TechCrunch article: http://techcrunch.com/2013/03/12/users-have-low-tolerance-for-buggy-apps-only-16-will-try-a-failing-app-more-than-twice/Trash can by Yellow Icon Design (http://www.yellowicon.com/) under GPU General Public License
  • Picture courtesy of http://www.flickr.com/photos/mulad/ (Creative Commons Attribution 2.0 Generic)
  • Suit image by http://www.flickr.com/photos/charlesfettinger/ (Creative Commons Attribution 2.0 Generic)
  • Picture courtesy of http://www.flickr.com/photos/wwarby/ (Creative Commons Attribution 2.0 Generic)
  • Wall image by http://www.flickr.com/photos/easylocum/ (Creative Commons Attribution 2.0 Generic)
  • Wall image by http://www.flickr.com/photos/riebart/ (Creative Commons Attribution 2.0 Generic)
  • Wall image by http://www.flickr.com/photos/zomgitsbrian/ (Creative Commons Attribution 2.0 Generic)
  • Wall image by http://www.flickr.com/photos/andreweason/ (Creative Commons Attribution 2.0 Generic)
  • Data image by http://www.flickr.com/photos/tensafefrogs/ (Creative Commons Attribution 2.0 Generic)
  • Wall image by http://www.flickr.com/photos/riebart/ (Creative Commons Attribution 2.0 Generic)
  • Wall image by http://www.flickr.com/photos/carbonnyc/ (Creative Commons Attribution 2.0 Generic)
  • Security image by http://www.flickr.com/photos/zigazou76/ (Creative Commons Attribution 2.0 Generic)
  • http://developer.android.com/tools/help/monkey.html
  • http://developer.android.com/tools/testing/testing_android.html
  • http://robolectric.org
  • http://developer.android.com/tools/testing/testing_ui.html
  • https://code.google.com/p/android-test-kit/
  • https://code.google.com/p/robotium/
  • http://appium.io/
  • https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html
  • https://developer.apple.com/library/ios/documentation/ToolsLanguages/Conceptual/Xcode_Overview/UnitTestYourApp/UnitTestYourApp.html
  • https://github.com/kif-framework/KIF
  • http://square.github.io/spoon/
  • https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/xcode_guide-continuous_integration/000-About_Continuous_Integration/about_continuous_integration.html#//apple_ref/doc/uid/TP40013292
  • Chains image by http://www.flickr.com/photos/colinsd40/ (Creative Commons Attribution 2.0 Generic)
  • Surgical tools image by http://www.flickr.com/photos/wien/ (Creative Commons Attribution 2.0 Generic)
  • https://appthwack.com
  • When & How to Successfully use Test Automation for Mobile Applications

    1. 1. Testing for Android & iOS When, Where, and How to Successfully Use Test Automation Trent Peterson | @tdpeterson | trent@appthwack.com
    2. 2. Who?
    3. 3. AGENDA What is testing? What’s the problem? Why should I care? When: Choosing our battles Where: Best Practices How: Tools & Frameworks Q&A
    4. 4. INTRO TO TESTING
    5. 5. Types of tests • • • • • • • • Functional Non-functional Integration Unit Regression Smoke Usability Performance • Stress • Acceptance • Load • Sanity • End-to-End • System • Security And on and on…
    6. 6. Testing distilled for this discussion Functional Testing Non-functional Testing
    7. 7. THE PROBLEM
    8. 8. Building things is hard.
    9. 9. WHY SHOULD I CARE?
    10. 10. So what’s the big deal? 1,000,000+ apps in 1,000,000+ apps in 2 crashes and 84% will uninstall - Compuware as reported in TechCrunch
    11. 11. We’ve been here before.
    12. 12. CHOOSING OUR BATTLES
    13. 13. First, what’s our goal? Save time! BE REALISTIC Save money! • Increase regularity of testing Shrink QA! • Free up resources for other testing efforts • Enable previously impossible tests AUTOMATE EVERYTHING !
    14. 14. Deciding what to automate How often will I test this? Is this something a human is good at? How much effort will automation require?
    15. 15. Some examples AUTOMATE MANUAL • Granular functionality (unit tests) • Repetitive tasks (update paths, navigation, etc.) • Performance • UX • Responsiveness and “feel” • New functionality
    16. 16. Our options (Machine) JVM Emulators Real Devices • FAST • Unit tests only* • Mocks • Cheap • Catch layout issues • Simulate calls/SMS • SLOW • Illusion of 1:1 with real devices • Exactly what end customers use • Performance data • Fast • Expensive: Purchase, maintai n, etc.
    17. 17. Our options (Human) QA Crowd End Users • Pros • Real feedback • Know the product • Expensive • Real feedback • Might know product • Expensive (usually) • Slow • Don’t do this.
    18. 18. BEST PRACTICES
    19. 19. No barriers between Dev & QA
    20. 20. Automation is a software project
    21. 21. Automation is a tool, not a solution
    22. 22. Automate with precision
    23. 23. It’s all about data
    24. 24. Know your matrix
    25. 25. Abstraction is important …but don’t obsess
    26. 26. TOOLS and FRAMEWORKS
    27. 27. Tool: Recorders GOOD BAD • Fast • Non-developers can create tests • Fragile • Difficult to maintain • Non-developers can create tests BEST PRACTICE • Treat as a fancy spy
    28. 28. Tool: UI Exerciser Monkey $ adb shell monkey –p my.sweet.app –v 1000
    29. 29. Framework: Android Testing SUMMARY • JUnit/Java • Instrumentation provides hooks into Android SDK to control Android-specific functionality • Very basic functionality supported • The foundation for most Android testing. Learn it, use it, love it.
    30. 30. Framework: Robolectric SUMMARY • JUnit • Runs in JVM • Mocks Android functionality with “shadow” objects • Super fast • Not everything is mocked (also, mocks) • Great for staying sane while developing
    31. 31. Framework: UI Automator SUMMARY • JUnit/Java • Extends testing framework with UI selectors and manipulators • Handle/control system dialogs, etc • 4.1+ only • Excellent choice if you have the luxury of supporting 4.1+.
    32. 32. Framework: Espresso SUMMARY • JUnit/Java • Extends testing framework with UI selectors and manipulators • Removes boilerplate code (sleeps, etc) • Very new and unstable • If you’re feeling adventurous give it a try
    33. 33. Framework: Robotium SUMMARY • JUnit extension • Runs on emulators and real devices • Oriented towards black-box testing and ensuring real-world outcomes • Limited to the app under test • Great choice for developers going beyond basic unit testing and testing UI.
    34. 34. Framework: Cucumber-based SUMMARY • Human-readable tests • Runs on emulators and real devices • Cross-platform • Complex scenarios require development • Behavior-driven design. Be careful! • Great choice for simple flows and teams with limited developer resources.
    35. 35. Framework: Appium SUMMARY • Supports many language adapters • Runs on emulators and real devices • Cross-platform • New, and not all WebDriver concepts map intuitively to native apps • Great choice for teams happy with Selenium and expanding to native.
    36. 36. Framework: UI Automation SUMMARY • • • • JavaScript Installs with Xcode Functions with other instruments Has a recorder (Careful) • Dev/maintenance of JS • Good choice to get basic tests up and running quickly.
    37. 37. Framework: XCTest SUMMARY • Objective C • Installs with Xcode • Poor documentation • Obvious choice for unit testing.
    38. 38. Framework: KIF SUMMARY • Objective C • Enables functional test automation capabilities by providing UI helpers • Relies on undocumented APIs • Great choice for devs automating functional tests.
    39. 39. Tool: Spoon Spoon Instrumentation Tests
    40. 40. Tool: Bots Continuous Integration Execute tests, etc.
    41. 41. Frameworks: Commercial GOOD BAD • Support • Added features and integrations • Longevity? • Proprietary • Often heavy and difficult to maintain • $$$ ADVICE • Evaluate very carefully and acknowledge lock-in.
    42. 42. When choosing, ask… What are we testing? What are our automation goals? Who will write the automated tests?
    43. 43. SHAMELESS PLUG
    44. 44. AppThwack • • • • 100s of non-rooted devices Results in minutes Built-in compatibility tests Supports all popular automation frameworks • Simple API and integrations • Focused, comprehensive reports
    45. 45. Q&A Trent Peterson | @tdpeterson | trent@appthwack.com
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×