Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mobile Test Automation

1,542 views

Published on

How to use cloud-based app testing to ensure that your mobile app doesn't suffer from bad reviews!

After an overview of the current state of mobile devices (operating system and device fragmentation for Android, iOS and Windows Phone), we take a look at current scientific studies that analyze app store reviews on a large scale, in order to extract useful app quality data.

In order to improve your app's quality, you will have to test a lot. Manual testing is usually limited to a few devices. Therefore, the best solution is often using cloud-based services like the Xamarin Test Cloud, which can test your app with a single click on 1000+ real devices.

You will also learn the basics of setting up your app for cloud-based testing and what kind of data you can extract.

Published in: Mobile
  • Be the first to comment

Mobile Test Automation

  1. 1. Mobile Test Automation Andreas Jakl @andijakl andreas.jakl@tieto.com Mobile Developer After-Work #7 – Xamarin: April 9th, 2015
  2. 2. Andreas Jakl Mobility Evangelist, Tieto – @andijakl – andreas.jakl@tieto.com History – Mopius: Startup-Founder (NFC & Mobile Apps) – Nokia, Finland: Technology Wizard – FH Hagenberg: Assistant Professor Mobile Computing – Siemens / BenQ Mobile, Germany: Augmented Reality-Apps Mobile Test Automation - Andreas Jakl2
  3. 3. TESTING – WHY? Versions & More Mobile Test Automation - Andreas Jakl3
  4. 4. Slow uptake of new versions Manufacturer- modified UIs Countless screen sizes & densities Mobile Test Automation - Andreas Jakl4 https://developer.android.com/about/dashboards/index.html http://stats.unity3d.com/mobile/os-android.html
  5. 5. Fast adoption of new OS – Design updates – API & architecture changes (e.g., 64 bit enforcement) Quick update of apps required Same OS version, but many iOS devices in use Mobile Test Automation - Andreas Jakl5 https://developer.apple.com/support/appstore/ http://stats.unity3d.com/mobile/device-ios.html
  6. 6. Similar hardware & software base – 512 MB vs 1+ GB memory – Processor speed – 4 supported screen resolutions Relevant: v8(.1) > 90% – But: Windows 10 upcoming • More resolutions, etc. Mobile Test Automation - Andreas Jakl6 http://blogs.windows.com/buildingapps/2014/12/22/windows-and-windows-phone-store-trends-december-2014-update/ http://blog.adduplex.com/2015/03/adduplex-windows-phone-statistics.html
  7. 7. IMPACT OF TESTING Successful? Mobile Test Automation - Andreas Jakl7
  8. 8. Reviews vs Downloads Review count & rating correlate to downloads Mobile Test Automation - Andreas Jakl8 Bin Lu, et al.: Why People Hate Your App – Making Sense of User Feedback in a Mobile App Store. Carnegie Mellon University, 2013 Data from 13 million user reviews, 171,439 apps Average rating: 3.9 (standard deviation: 1.48) 0 2 000 000 4 000 000 6 000 000 8 000 000
  9. 9. Review Analysis Analysis – Most frequent complaints • Functional errors • Feature requests • App crashes Mobile Test Automation - Andreas Jakl9 Claudia Iacob, Rachel Harrison: “Retrieving and Analysing Mobile Apps Feature Requests from Online Reviews”, Oxford Brookes University, 2013 Kahlid. H, et al.: “What Do Mobile App Users Complain About? A Study on Free iOS Apps”, Queen’s University, Kingston, 2014 Gunwoong Leea, T. S. Raghua: “Determinants of Mobile Apps' Success: Evidence from the App Store Market”, 2014
  10. 10. Review Analysis Analysis – Most negative impact • Privacy and ethical issues • Hidden app costs Mobile Test Automation - Andreas Jakl10 Claudia Iacob, Rachel Harrison: “Retrieving and Analysing Mobile Apps Feature Requests from Online Reviews”, Oxford Brookes University, 2013 Kahlid. H, et al.: “What Do Mobile App Users Complain About? A Study on Free iOS Apps”, Queen’s University, Kingston, 2014 Gunwoong Leea, T. S. Raghua: “Determinants of Mobile Apps' Success: Evidence from the App Store Market”, 2014
  11. 11. Customer Retention Quality (feature) upgrades – Up to 3x improvement in survival rates Mobile Test Automation - Andreas Jakl11 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 1 Month after Download 6 Months after Download 1 Year after Download Customer Retention in Mobile Apps
  12. 12. TESTING STRATEGIES Manual Mobile Test Automation - Andreas Jakl12
  13. 13. Capturing Crash Data Mobile Test Automation - Andreas Jakl13
  14. 14. Monitoring User Reviews Mobile Test Automation - Andreas Jakl14 Bin Lu, et al.: Why People Hate Your App – Making Sense of User Feedback in a Mobile App Store. Carnegie Mellon University, 2013
  15. 15. Static Code Analysis Mobile Test Automation - Andreas Jakl15
  16. 16. Emulator / Simulator Mobile Test Automation - Andreas Jakl16 Emulator Emulates real hardware + more accurate – slow Android Simulator Simulates device behavior + faster – less accurate iOS / Windows
  17. 17. Reliable? Mobile Test Automation - Andreas Jakl17 0%of your users will run your app on an emulator
  18. 18. Own Devices Mobile Test Automation - Andreas Jakl18 How many test devices do you have?
  19. 19. Open Device Lab Vienna Mobile Test Automation - Andreas Jakl19 Free testing on real devices http://www.open-device-lab-vienna.at/
  20. 20. Automated UI Tests Mobile Test Automation - Andreas Jakl20 How many tests can you run per hour / build / release? Number of builds Cost Automated tests Manual tests
  21. 21. TEST CLOUD Automated Testing on 1000s of Devices Mobile Test Automation - Andreas Jakl21
  22. 22. Xamarin Test Cloud Automated tests – UI interaction – Device features (GPS, etc) – Performance 1,300+ devices – Android & iOS – testcloud.xamarin.com/devices Mobile Test Automation - Andreas Jakl22
  23. 23. Workflow Mobile Test Automation - Andreas Jakl23
  24. 24. + Continuous Integration Mobile Test Automation - Andreas Jakl24
  25. 25. Calabash Xamarin.UiTest Test Frameworks Mobile Test Automation - Andreas Jakl25 Write tests in Ruby C# Run with Gherkin NUnit
  26. 26. Executing Tests Mobile Test Automation - Andreas Jakl26
  27. 27. Process 1. Write Tests 2. Setup Project for Testing 3. Test Locally 4. Create app on Xamarin Test Cloud 5. Upload & Run on Cloud via Command Line Mobile Test Automation - Andreas Jakl27
  28. 28. Writing Tests: NUnit Mobile Test Automation - Andreas Jakl28 Generic tests possible with Xamarin.Forms
  29. 29. Test Setup Mobile Test Automation - Andreas Jakl29 Platform-specific initialization & test differences
  30. 30. Setup Project Mobile Test Automation - Andreas Jakl30 Disable shared Mono runtime
  31. 31. API Key Mobile Test Automation - Andreas Jakl31 Create team & app to get unique API key
  32. 32. Run Tests Locally Mobile Test Automation - Andreas Jakl32 Test on your devices first
  33. 33. Mobile Test Automation - Andreas Jakl33 Cloud Setup Choose devices
  34. 34. Mobile Test Automation - Andreas Jakl34 Cloud Setup System settings
  35. 35. Mobile Test Automation - Andreas Jakl35 Running Tests in the cloud Command line only
  36. 36. Mobile Test Automation - Andreas Jakl36 Test results Dashboard
  37. 37. Mobile Test Automation - Andreas Jakl37 Test results Screenshots
  38. 38. Custom Screenshots Take & describe screenshots Enable local screenshots Mobile Test Automation - Andreas Jakl38 [TestCase (TestName = "Click button and show success message")] public void TestCaseSuccess () { AppResult[] result = _app.Query(InitialMessage); Assert.IsTrue(result.Any(), "The initial message string isn't correct"); _app.Screenshot ("Initial screen"); _app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY"). EnableLocalScreenshots().StartApp();
  39. 39. Custom Screenshots Take & describe screenshots Enable local screenshots Mobile Test Automation - Andreas Jakl39 [TestCase (TestName = "Click button and show success message")] public void TestCaseSuccess () { AppResult[] result = _app.Query(InitialMessage); Assert.IsTrue(result.Any(), "The initial message string isn't correct"); _app.Screenshot ("Initial screen"); _app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY"). EnableLocalScreenshots().StartApp();
  40. 40. Testing Finished App Packages Find IDs of UI elements Mobile Test Automation - Andreas Jakl40 if (TestEnvironment.Platform.Equals(TestPlatform.Local)) { _app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY"). EnableLocalScreenshots().StartApp(); _app.Repl (); // In Bash, enter app.Query(c=>c.Button()) or c.All() }
  41. 41. Testing Finished App Packages Find IDs of UI elements Mobile Test Automation - Andreas Jakl41 if (TestEnvironment.Platform.Equals(TestPlatform.Local)) { _app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY"). EnableLocalScreenshots().StartApp(); _app.Repl (); // In Bash, enter app.Query(c=>c.All()) }
  42. 42. Alternatives Testdroid – testdroid.com/ Perfecto Mobile – perfectomobile.com/solution/test-automation Mobile Test Automation - Andreas Jakl42
  43. 43. Thank You! Andreas Jakl @andijakl andreas.jakl@tieto.com Mobile Test Automation - Andreas Jakl43

×