The Basics of Mobile App Testing & Test Automation

2,994 views
2,819 views

Published on

Watch a live presentation at http://bit.ly/1gq9qR6

The name of the game in today’s competitive mobile application landscape is a speed combined with robustness. How can you quickly deliver the best quality apps and how to ensure your app will stay on the top of download rankings at app markets? These questions will be answered as well as a comprehensive approach to improve your development process and app quality will be presented.

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,994
On SlideShare
0
From Embeds
0
Number of Embeds
58
Actions
Shares
0
Downloads
131
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

The Basics of Mobile App Testing & Test Automation

  1. 1. The Basics of Mobile App Testing & Test Automation 26 March 2014 Ville-Veikko Helppi Technical Product Manager ville-veikko.helppi@bitbar.com
  2. 2. Agenda • App Quality and Mobile App Testing • How To Get Started with Test Automation? • Test Automation Frameworks • 10 Tips for Successful App Testing • Testdroid Update • Demonstration • Q&A © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 2
  3. 3. Agenda • App Quality and Mobile App Testing • How To Get Started with Test Automation? • Test Automation Frameworks • 10 Tips for Successful App Testing • Testdroid Update • Demonstration • Q&A © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 3
  4. 4. Challenges in App Development © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 4 Fact: Most typical application fails when tested across different device variants used by consumers • Failures: Both SW and HW cause failures • Emulators do not offer realistic platform for dev • Even app installation fails – not tested at all • Application crashes, leaks memory, consumes too much resources, behaves not as meant to behave • UI scaling/display/resolution – common problem • Application performance is bad, draining battery
  5. 5. Why Apps Fail? © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 5
  6. 6. Testdroid Research Available Soon… © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 6
  7. 7. What Improves the App Quality? • Automate generic things as much as you can • During the development your app changes – the same must apply for testing! • Carefully select (testing) technology & partner • Use all 24 hours per day with test automation • Cloud-based platform is the only solution to get you quickly covered in the global markets © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 7
  8. 8. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel • Testdroid has helped thousands of app developers in these subverticals! • What are the critical elements in each these verticals? © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 8
  9. 9. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel • User Experience! • Resource consumption & validation (CPU, Mem…) • Fully utilizing hardware – e.g. Touch screen • Mobile games require access to graphics APIs (e.g. OpenGL ES) © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 9
  10. 10. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel • Usability! • Functional and Behavioral testing • Metrics analysis for all captured data (e.g. logs, screenshots, perf stats) • Connection with Social Media + other apps© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 10
  11. 11. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel • Performance! • Connectivity, robustness and durability • Screen orientation (portrait vs. landscape) • Graphics quality, streaming capabilities • User profiles© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 11
  12. 12. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel • Security! • Secure connectivity with back-end systems • Top Quality – extremely brand sensitive vertical • Compliances and verification between real devices and infrastructure © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 12
  13. 13. Mobile Apps for Different Verticals Mobile Games Utilities & Tools Video Streaming & Multimedia Banking & Payment Retail & Travel • Data! • Connectivity and data connection with back-end systems • Configurability of the app • Bad quality can hurt the brand and leave you without customers © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 13
  14. 14. Agenda • App Quality and Mobile App Testing • How To Get Started with Test Automation? • Test Automation Frameworks • 10 Tips for Successful App Testing • Testdroid Update • Demonstration • Q&A © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 14
  15. 15. Prerequisites for Mobile App Testing • Testing environment as authentic as possible – Real devices <-> emulators/simulators • Test coverage needs to be adequate – Software is not identical in all Android/iOS devices – Hardware is even more fragmented landscape • Surrounding testing conditions (e.g. network) • Try not to build in repetitive, manual processes • Test automation framework that works for you © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 15
  16. 16. Trade-offs – Manual vs. Automation • Costs & Assets – Time, People, Infrastructure, Tools, Training • App Development & Testing Cycle – Together – Testing after Daily Builds (Continuous Integration) • Tools & Technology – Frameworks – Myriad of frameworks available to help you • Test Coverage & Reusability – Global coverage is CRITICAL to SUCCESS • Outcome – Efficiency, Effectiveness and Results © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 16
  17. 17. Manual Automation Smaller coverage, More money burnt & time wasted, Error-prone Large coverage, quick ly completed, Less money & time wasted, Exact results. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 17
  18. 18. What Framework Works You The Best? • High-level & declarative test automation frameworks work well when app is under dev • Easy to integrate with continuous integration • Distance between use case and actual test • What do you build: Native, Hybrid or Web app? • The technical competence in your organization • Desired outcome from test automation © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 18
  19. 19. Different Ways of Doing Test Automation Automatic test exercisersRecord and PlaybackHand written test scripts Benefits: Accurate, specific to your testing needs, plenty of options with frameworks, tools Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools avail’ty Fastest & extremely automated, excellent for smoke testing/quick testing, availability Tradeoffs: Takes a lot of time, ties resources to write test cases/scripts, error- prone (humans) Compelling Recorder+Playback tools available for only few test automation frameworks Not accurate as real test cases © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 19
  20. 20. Developers Source code Repository Build failed Successful build QA & Testers Error reporting Test Cases Application Users / Consumers Commit © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 20 Test Automation During the Dev
  21. 21. Developers Source code Repository Build failed Successful build QA & Testers Error reporting Test Cases Application Users / Consumers Commit © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 21 Test Automation During the Dev
  22. 22. Why Real Devices are Must-to-Have? • Emulators/simulators won’t help you to test… – User Experience and Usability – Hardware – Software – Infrastructure 0 % = the percentage of your app users that use emulator to run your app! © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 22
  23. 23. Test Automation Will Deliver You… • App Compatibility & Global coverage • Reusability • Test efficiency & effectiveness • Faster time-to-market Test Automation during the Mobile App Development Process Testrequirements/Planning Smoke testing Regression testing Regression testing Regression testing Regression testing Functional & Interoperability testing Functional & Interoperability testing Functional & Interoperability testing Performance & Stress testing Performance & Stress testing Compatibility testing © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 23
  24. 24. Agenda • App Quality and Mobile App Testing • How To Get Started with Test Automation? • Test Automation Frameworks • 10 Tips for Successful App Testing • Testdroid Update • Demonstration • Q&A © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 24
  25. 25. Android Instrumentation Framework • InstrumentationTestRunner – the primary plumbing for running tests on Android • Android Instrumentation Framework is built on top of JUnit – a standard test framework on for any Java development • Mock objects – methods for creating mock system objects such as content, service and intent © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 25
  26. 26. iOS Instruments (with Xcode) © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 26
  27. 27. Android Example: The Family Tree of Test Automation Frameworks JUnit Android Instrumentation Framework Robotium Espresso UI Automator Calabash Appium © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 27
  28. 28. Comparison of Test Automation Frameworks Supports Calabash Robotium uiautomator Espresso Appium Android Yes Yes Yes Yes Yes iOS Yes No No No Yes Mobile web Hybrid (webviews) Yes (Android) Limited to x.y clicks No Yes (Android & iOS) Scripting Language Ruby Java Java Java Almost any Test creation tools CLI Testdroid Recorder UI Automator viewer Hierarchy Viewer Appium.app Supported API levels All All 16 => 8, 10, 15-19 All Community Active Contributors Google Google Active © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 28
  29. 29. Agenda • App Quality and Mobile App Testing • How To Get Started with Test Automation? • Test Automation Frameworks • 10 Tips for Successful App Testing • Testdroid Update • Demonstration • Q&A © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 29
  30. 30. Tip #1: Test early, Test often © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 30
  31. 31. Tip #2: Plan What to Automate People Infrastructure Tools Training Time MANUAL AUTOMATED © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 31
  32. 32. Tip #3: Use only Real Devices © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 32
  33. 33. Tip #4: Use Atomic Test Units Test iteration #1 Test iteration #2 © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 33
  34. 34. Tip #5: Create Hermetic Tests © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 34
  35. 35. Tip #6: Use All Possible Devices Testdroid Cloud’s 300+ Android devices = 93-95% global Android volumes! © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 35
  36. 36. Tip #7: Separate Apps and Tests Your App © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 36
  37. 37. Tip #8: Output Everything to Logs © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 37
  38. 38. Tip #9: End-to-End Testing © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 38
  39. 39. Tip #10: Integrate Dev & Testing Commit Developers Source code Repository Build failed Successful build QA & Testers Error reporting Test Cases Application Users / Consumers © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 39
  40. 40. Agenda • App Quality and Mobile App Testing • How To Get Started with Test Automation? • Test Automation Frameworks • 10 Tips for Successful App Testing • Testdroid Update • Demonstration • Q&A © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 40
  41. 41. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 41
  42. 42. Agenda • App Quality and Mobile App Testing • How To Get Started with Test Automation? • Test Automation Frameworks • 10 Tips for Successful App Testing • Testdroid Update • Demonstration • Q&A © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 42
  43. 43. Agenda • App Quality and Mobile App Testing • How To Get Started with Test Automation? • Test Automation Frameworks • 10 Tips for Successful App Testing • Testdroid Update • Demonstration • Q&A © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 43
  44. 44. © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 44

×