1.
Mobile Test Automation
Andreas Jakl
@andijakl
andreas.jakl@tieto.com
Mobile Developer After-Work #7 – Xamarin: April 9th, 2015

2.
Andreas Jakl
Mobility Evangelist, Tieto
– @andijakl
– andreas.jakl@tieto.com
History
– Mopius: Startup-Founder (NFC & Mobile Apps)
– Nokia, Finland: Technology Wizard
– FH Hagenberg: Assistant Professor Mobile Computing
– Siemens / BenQ Mobile, Germany: Augmented Reality-Apps
Mobile Test Automation - Andreas Jakl2

3.
TESTING – WHY?
Versions & More
Mobile Test Automation - Andreas Jakl3

4.
Slow uptake of new
versions
Manufacturer-
modified UIs
Countless screen
sizes & densities
Mobile Test Automation - Andreas Jakl4
https://developer.android.com/about/dashboards/index.html
http://stats.unity3d.com/mobile/os-android.html

5.
Fast adoption of new OS
– Design updates
– API & architecture changes
(e.g., 64 bit enforcement)
Quick update of apps required
Same OS version, but many iOS
devices in use
Mobile Test Automation - Andreas Jakl5
https://developer.apple.com/support/appstore/
http://stats.unity3d.com/mobile/device-ios.html

6.
Similar hardware & software base
– 512 MB vs 1+ GB memory
– Processor speed
– 4 supported screen resolutions
Relevant: v8(.1) > 90%
– But: Windows 10 upcoming
• More resolutions, etc.
Mobile Test Automation - Andreas Jakl6
http://blogs.windows.com/buildingapps/2014/12/22/windows-and-windows-phone-store-trends-december-2014-update/
http://blog.adduplex.com/2015/03/adduplex-windows-phone-statistics.html

7.
IMPACT OF TESTING
Successful?
Mobile Test Automation - Andreas Jakl7

8.
Reviews vs Downloads
Review count & rating correlate to downloads
Mobile Test Automation - Andreas Jakl8
Bin Lu, et al.: Why People Hate Your App – Making Sense of User Feedback in a Mobile App Store.
Carnegie Mellon University, 2013
Data from 13 million user reviews, 171,439 apps
Average rating: 3.9
(standard deviation: 1.48)
0
2 000 000
4 000 000
6 000 000
8 000 000

9.
Review Analysis
Analysis
– Most frequent complaints
• Functional errors
• Feature requests
• App crashes
Mobile Test Automation - Andreas Jakl9
Claudia Iacob, Rachel Harrison: “Retrieving and Analysing Mobile Apps Feature Requests from Online Reviews”, Oxford Brookes University, 2013
Kahlid. H, et al.: “What Do Mobile App Users Complain About? A Study on Free iOS Apps”, Queen’s University, Kingston, 2014
Gunwoong Leea, T. S. Raghua: “Determinants of Mobile Apps' Success: Evidence from the App Store Market”, 2014

10.
Review Analysis
Analysis
– Most negative impact
• Privacy and ethical issues
• Hidden app costs
Mobile Test Automation - Andreas Jakl10
Claudia Iacob, Rachel Harrison: “Retrieving and Analysing Mobile Apps Feature Requests from Online Reviews”, Oxford Brookes University, 2013
Kahlid. H, et al.: “What Do Mobile App Users Complain About? A Study on Free iOS Apps”, Queen’s University, Kingston, 2014
Gunwoong Leea, T. S. Raghua: “Determinants of Mobile Apps' Success: Evidence from the App Store Market”, 2014

11.
Customer Retention
Quality (feature)
upgrades
– Up to 3x
improvement in
survival rates
Mobile Test Automation - Andreas Jakl11
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
1 Month after Download 6 Months after Download 1 Year after Download
Customer Retention in Mobile Apps

12.
TESTING STRATEGIES
Manual
Mobile Test Automation - Andreas Jakl12

13.
Capturing Crash Data
Mobile Test Automation - Andreas Jakl13

14.
Monitoring User Reviews
Mobile Test Automation - Andreas Jakl14
Bin Lu, et al.: Why People Hate Your App – Making Sense of User Feedback in a Mobile App Store.
Carnegie Mellon University, 2013

15.
Static Code Analysis
Mobile Test Automation - Andreas Jakl15

16.
Emulator / Simulator
Mobile Test Automation - Andreas Jakl16
Emulator
Emulates real
hardware
+ more accurate
– slow
Android
Simulator
Simulates device
behavior
+ faster
– less accurate
iOS / Windows

17.
Reliable?
Mobile Test Automation - Andreas Jakl17
0%of your users will run your app on an emulator

18.
Own Devices
Mobile Test Automation - Andreas Jakl18
How many test devices do you have?

19.
Open Device Lab Vienna
Mobile Test Automation - Andreas Jakl19
Free testing on real devices
http://www.open-device-lab-vienna.at/

20.
Automated UI Tests
Mobile Test Automation - Andreas Jakl20
How many tests can you run per hour / build / release?
Number of builds
Cost
Automated tests
Manual tests

21.
TEST CLOUD
Automated Testing on 1000s of Devices
Mobile Test Automation - Andreas Jakl21

22.
Xamarin Test Cloud
Automated tests
– UI interaction
– Device features (GPS, etc)
– Performance
1,300+ devices
– Android & iOS
– testcloud.xamarin.com/devices
Mobile Test Automation - Andreas Jakl22

23.
Workflow
Mobile Test Automation - Andreas Jakl23

24.
+ Continuous Integration
Mobile Test Automation - Andreas Jakl24

25.
Calabash Xamarin.UiTest
Test Frameworks
Mobile Test Automation - Andreas Jakl25
Write tests in Ruby C#
Run with Gherkin NUnit

26.
Executing Tests
Mobile Test Automation - Andreas Jakl26

27.
Process
1. Write Tests
2. Setup Project for Testing
3. Test Locally
4. Create app on Xamarin Test Cloud
5. Upload & Run on Cloud via Command Line
Mobile Test Automation - Andreas Jakl27

28.
Writing Tests: NUnit
Mobile Test Automation - Andreas Jakl28
Generic tests possible
with Xamarin.Forms

29.
Test Setup
Mobile Test Automation - Andreas Jakl29
Platform-specific
initialization & test
differences

30.
Setup Project
Mobile Test Automation - Andreas Jakl30
Disable shared
Mono runtime

31.
API Key
Mobile Test Automation - Andreas Jakl31
Create team & app to
get unique API key

32.
Run Tests
Locally
Mobile Test Automation - Andreas Jakl32
Test on your
devices first

33.
Mobile Test Automation - Andreas Jakl33
Cloud Setup
Choose devices

34.
Mobile Test Automation - Andreas Jakl34
Cloud Setup
System settings

35.
Mobile Test Automation - Andreas Jakl35
Running Tests
in the cloud
Command line
only

36.
Mobile Test Automation - Andreas Jakl36
Test results
Dashboard

37.
Mobile Test Automation - Andreas Jakl37
Test results
Screenshots

38.
Custom Screenshots
Take & describe screenshots
Enable local screenshots
Mobile Test Automation - Andreas Jakl38
[TestCase (TestName = "Click button and show success message")]
public void TestCaseSuccess ()
{
AppResult[] result = _app.Query(InitialMessage);
Assert.IsTrue(result.Any(), "The initial message string isn't correct");
_app.Screenshot ("Initial screen");
_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").
EnableLocalScreenshots().StartApp();

39.
Custom Screenshots
Take & describe screenshots
Enable local screenshots
Mobile Test Automation - Andreas Jakl39
[TestCase (TestName = "Click button and show success message")]
public void TestCaseSuccess ()
{
AppResult[] result = _app.Query(InitialMessage);
Assert.IsTrue(result.Any(), "The initial message string isn't correct");
_app.Screenshot ("Initial screen");
_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").
EnableLocalScreenshots().StartApp();

40.
Testing Finished App Packages
Find IDs of UI elements
Mobile Test Automation - Andreas Jakl40
if (TestEnvironment.Platform.Equals(TestPlatform.Local))
{
_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").
EnableLocalScreenshots().StartApp();
_app.Repl ();
// In Bash, enter app.Query(c=>c.Button()) or c.All()
}

41.
Testing Finished App Packages
Find IDs of UI elements
Mobile Test Automation - Andreas Jakl41
if (TestEnvironment.Platform.Equals(TestPlatform.Local))
{
_app = ConfigureApp.Android.ApkFile(PathToAPK).ApiKey("YOUR_API_KEY").
EnableLocalScreenshots().StartApp();
_app.Repl ();
// In Bash, enter app.Query(c=>c.All())
}

42.
Alternatives
Testdroid
– testdroid.com/
Perfecto Mobile
– perfectomobile.com/solution/test-automation
Mobile Test Automation - Andreas Jakl42

43.
Thank You!
Andreas Jakl
@andijakl
andreas.jakl@tieto.com
Mobile Test Automation - Andreas Jakl43