Mobile Test Automation

Mobile Test Automation
Andreas Jakl
@andijakl
andreas.jakl@tieto.com
Mobile Developer After-Work #7 – Xamarin: April 9th, 2015

Andreas Jakl
Mobility Evangelist, Tieto
– @andijakl
– andreas.jakl@tieto.com
History
– Mopius: Startup-Founder (NFC & Mobile Apps)
– Nokia, Finland: Technology Wizard
– FH Hagenberg: Assistant Professor Mobile Computing
– Siemens / BenQ Mobile, Germany: Augmented Reality-Apps
Mobile Test Automation - Andreas Jakl2

TESTING – WHY?
Versions & More

Slow uptake of new
versions
Manufacturer-
modified UIs
Countless screen
sizes & densities
https://developer.android.com/about/dashboards/index.html
http://stats.unity3d.com/mobile/os-android.html

Fast adoption of new OS
– Design updates
– API & architecture changes
(e.g., 64 bit enforcement)
Quick update of apps required
Same OS version, but many iOS
devices in use
https://developer.apple.com/support/appstore/
http://stats.unity3d.com/mobile/device-ios.html

Similar hardware & software base
– 512 MB vs 1+ GB memory
– Processor speed
– 4 supported screen resolutions
Relevant: v8(.1) > 90%
– But: Windows 10 upcoming
• More resolutions, etc.
http://blogs.windows.com/buildingapps/2014/12/22/windows-and-windows-phone-store-trends-december-2014-update/
http://blog.adduplex.com/2015/03/adduplex-windows-phone-statistics.html

IMPACT OF TESTING
Successful?

Reviews vs Downloads
Review count & rating correlate to downloads
Bin Lu, et al.: Why People Hate Your App – Making Sense of User Feedback in a Mobile App Store.
Carnegie Mellon University, 2013
Data from 13 million user reviews, 171,439 apps
Average rating: 3.9
(standard deviation: 1.48)
0
2 000 000
4 000 000
6 000 000
8 000 000

Review Analysis
Analysis
– Most frequent complaints
• Functional errors
• Feature requests
• App crashes
Claudia Iacob, Rachel Harrison: “Retrieving and Analysing Mobile Apps Feature Requests from Online Reviews”, Oxford Brookes University, 2013
Kahlid. H, et al.: “What Do Mobile App Users Complain About? A Study on Free iOS Apps”, Queen’s University, Kingston, 2014
Gunwoong Leea, T. S. Raghua: “Determinants of Mobile Apps' Success: Evidence from the App Store Market”, 2014

Review Analysis
Analysis
– Most negative impact
• Privacy and ethical issues
• Hidden app costs
Claudia Iacob, Rachel Harrison: “Retrieving and Analysing Mobile Apps Feature Requests from Online Reviews”, Oxford Brookes University, 2013
Kahlid. H, et al.: “What Do Mobile App Users Complain About? A Study on Free iOS Apps”, Queen’s University, Kingston, 2014
Gunwoong Leea, T. S. Raghua: “Determinants of Mobile Apps' Success: Evidence from the App Store Market”, 2014

Customer Retention
Quality (feature)
upgrades
– Up to 3x
improvement in
survival rates
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
1 Month after Download 6 Months after Download 1 Year after Download
Customer Retention in Mobile Apps

TESTING STRATEGIES
Manual

Capturing Crash Data

Monitoring User Reviews
Bin Lu, et al.: Why People Hate Your App – Making Sense of User Feedback in a Mobile App Store.
Carnegie Mellon University, 2013

Static Code Analysis

Emulator / Simulator
Emulator
Emulates real
hardware
+ more accurate
– slow
Android
Simulator
Simulates device
behavior
+ faster
– less accurate
iOS / Windows

Reliable?
0%of your users will run your app on an emulator

Own Devices
How many test devices do you have?

Open Device Lab Vienna
Free testing on real devices
http://www.open-device-lab-vienna.at/

Automated UI Tests
How many tests can you run per hour / build / release?
Number of builds
Cost
Automated tests
Manual tests

TEST CLOUD
Automated Testing on 1000s of Devices

Xamarin Test Cloud
Automated tests
– UI interaction
– Device features (GPS, etc)
– Performance
1,300+ devices
– Android & iOS
– testcloud.xamarin.com/devices

Workflow

+ Continuous Integration

Calabash Xamarin.UiTest
Test Frameworks
Write tests in Ruby C#
Run with Gherkin NUnit

Executing Tests

Process
1. Write Tests
2. Setup Project for Testing
3. Test Locally
4. Create app on Xamarin Test Cloud
5. Upload & Run on Cloud via Command Line

Writing Tests: NUnit
Generic tests possible
with Xamarin.Forms

Test Setup
Platform-specific
initialization & test
differences

Setup Project
Disable shared
Mono runtime

API Key
Create team & app to
get unique API key

Run Tests
Locally
Test on your
devices first

Cloud Setup
Choose devices

Cloud Setup
System settings

Running Tests
in the cloud
Command line
only

Test results
Dashboard

Test results
Screenshots

Custom Screenshots
Take & describe screenshots
Enable local screenshots
[TestCase (TestName = "Click button and show success message")]
public void TestCaseSuccess ()
{
AppResult[] result = _app.Query(InitialMessage);
Assert.IsTrue(result.Any(), "The initial message string isn't correct");
_app.Screenshot ("Initial screen");
_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").
EnableLocalScreenshots().StartApp();

Custom Screenshots
Take & describe screenshots
Enable local screenshots
[TestCase (TestName = "Click button and show success message")]
public void TestCaseSuccess ()
{
AppResult[] result = _app.Query(InitialMessage);
Assert.IsTrue(result.Any(), "The initial message string isn't correct");
_app.Screenshot ("Initial screen");

Testing Finished App Packages
Find IDs of UI elements
if (TestEnvironment.Platform.Equals(TestPlatform.Local))
{
_app.Repl ();
// In Bash, enter app.Query(c=>c.Button()) or c.All()
}

Testing Finished App Packages
Find IDs of UI elements
if (TestEnvironment.Platform.Equals(TestPlatform.Local))
{
_app.Repl ();
// In Bash, enter app.Query(c=>c.All())
}

Alternatives
Testdroid
– testdroid.com/
Perfecto Mobile
– perfectomobile.com/solution/test-automation

Thank You!
Andreas Jakl
@andijakl
andreas.jakl@tieto.com

Mobile Test Automation

More Related Content

What's hot

Viewers also liked

Similar to Mobile Test Automation

More from Andreas Jakl

Mobile Test Automation