This document discusses mobile end-to-end testing and exploratory testing tools. It covers the two categories of mobile automation technologies: instrumented and non-instrumented. It then discusses advantages of each technique and considerations for which to use. The document lists popular mobile automation tools for Android and iOS and outlines some pain points or challenges for testing on each platform. It concludes with proposing next steps to address the mobile testing "wall of pain".

1. MOET
https://github.com/eing/moet
Mobile End-to-End Testing
Mobile Exploratory Testing
Eing Ong
Intuit, Inc.
@eingong|eing.ong@intuit.com
1
1

2. Mobile'automation'technologies1
• Two categories
• Instrumented technique
• Non-instrumented technique
• What is instrumentation?
• Tests are compiled with the app
• Tests are installed & launched with the app
• Source code is required and may need to be modified
• Only one app can be executed at a time
• White box approach
2

3. Advantages'of'both'techniques1
Non$instrumentation- Instrumentation-
• Device'platform'agnostic1 • Elements'can'be'accessed1
• Test'code'reuse1 • Debugging'ease1
• Test'language'&'test'harness' • Test'verification'ease1
autonomy1 • Reduce'tools'dependencies1
• Support'for1 • Support'for1
Multi>applications'testing1 Installing'application1
Custom'composite'UI'elements1 Launching'application1
Database/server'API'assertions1 Cleanup'(kill'application)1
Use'of'external'libraries1 Test'execution'on'device1
'(e.g.'hardware/image'libraries)1 Code'coverage1
3

4. Which'technique'should'I'use'?1
Non-instrumentation Instrumentation
External3
interaction-
Hardware3 Text$based3
interaction- features-
Image3based3
features-
4

5. Mobile'automation'tools1
Mobile3- Non$ Instrumentation-
OS- instrumentation-
Android1 eggPlant,'Sikuli,'MOET,' Robotium,'Calabash,'
MonkeyRunner1 MonkeyTalk1
iOS1 eggPlant,'Sikuli,'MOET1 UIAutomation,'KIF,'
iCuke,'Frank,'UISpec,'
Zucchini,'Bwoken,'
Calabash,'MonkeyTalk,'
Applecart,'iOS>driver1
5

6. MOET1
Test
• Think design Login(“user1”,”passwd1”)
o Interfaces
o Creational pattern iPhone implementation
touch(100,100)
• Think reuse enter(username)
touch(100,200)
o Device independent tests
enter(password)
• Think One touch(150, 300)
o IDE
o Test harness and language iPhone Sikuli library
void enter()
void touch(x,y)
6

7. Anatomy'of''MOET1
Mobile3Application3Interface-
Device3Independent3Tests-
Runtime binding
Simulator/Device libraries
Android3application3 iPhone3application3
implementation- implementation-
Android3MonkeyRunner3
iPhone3Sikuli3Library-
Library-
7

8. Top Ten Wish List
a.k.a.
Mobile test automation
Wall of Pain
8
8

9. Android'Wall'of'Pain1
10. There’s more to test execution
o Command with report file generation :
adb shell am instrument -w -e reportFile junit-report.xml
com.intuit.test/com.zutubi.android.junitreport.JUnitReportTestRunner
o Date-time fix if using device snapshots :
adb shell date -s `date +"%Y%m%d.%H%M%S”`
9. Where are the test results ?
o Pulling results from device :
adb pull /data/data/com.intuit.test/files/junit-report.xml junit-
report.xml
9

10. Android'Wall'of'Pain1
10. There’s more to test execution
o Command with report file generation :
adb shell am instrument -w -e reportFile junit-report.xml
com.intuit.test/com.zutubi.android.junitreport.JUnitReportTestRunner
o Date-time fix if using device snapshots :
adb shell date -s `date +"%Y%m%d.%H%M%S”`
9. Where are the test results ?
o Pulling results from device :
adb pull /data/data/com.intuit.test/files/junit-report.xml junit-
report.xml
10

11. iOS'Wall'of'Pain1
8. iOS simulator…the singleton pattern
o Multiple instances of simulator
o Headless execution
7. What test harness ?
o Test grouping, test assertions, base test class
o Results needs to be displayable in browser
6. How about a CI plugin
o Test execution
o Code coverage
11

12. iOS'Wall'of'Pain1
8. iOS simulator…the singleton pattern
o Multiple instances of simulator
o Headless execution
7. What test harness ?
o Test grouping, test assertions, base test class
o Results needs to be displayable in browser
6. How about a CI plugin
o Test execution
o Code coverage
12

13. iOS'Wall'of'Pain1
8. iOS simulator…the singleton pattern
o Headless execution
o Multiple instances of simulator
7. What test harness ?
o Test grouping, test assertions, base test class
o Results needs to be displayable in browser
6. How about a CI plugin
o Test execution
o Test results
o Code coverage
13

14. iOS'Wall'of'Pain1
5. Where’s the IDE plugin ?
o Code formatting
o API completion
o Breakpoints, debugger
o “javadoc” help
o One click test execution of test method/testsuite
14

15. All'mobile'platforms1
4. Problem ? Search ?
o Blogs, stackoverflow, developer groups for devices, etc.
o Mobile testing one-stop tips & tricks (not questions)
o Community driven effort
Tip – Remove authorization prompt
Administrative rights
Update /etc/authorization
<key>system.privilege.taskport</key>
<dict>
<key>allow-root</key>
<!-- previous value <false/> -->
<true/>
Tip – Clean cache, preferences, SQLite
~/Library/Application Support/iPhone Simulator/{SDK}/Applications/<uuid>
Library: Preferences, Caches
Documents: sqlite3 <app>.sqlitedb
15

16. All'mobile'platforms1
3. How about a mobile test report ?
o Mobile test report standard e.g. mobile-test-report.xml
• Screenshots and video recording of failed tests
• Device logs
• Device detection
2. Wireless?
o OTA app installation
o Testing via wifi and carrier networks
o Selenium grid-like test infrastructure
16

17. All'mobile'platforms1
3. How about a mobile test report?
o Mobile test report standard e.g. mobile-test-report.xml
• Screenshots and video recording of failed tests
• Device logs
• Device detection
2. Mobile testing is anything but mobile
o OTA app installation and execution
o Testing via wifi and carrier networks
o Test tool independent component
o Selenium grid-like test infrastructure
17

18. All'mobile'platforms1
1. Yet another proprietary language?
o Open sourced test harness & language for
• Instrumented tests
• Non-instrumented tests
• Hybrid apps tests
18

19. Next'steps1
Eclipse mobile Test Test Test Continuous
testing plugin development execution reporting integration
Mobile3Application3Interface-
Device3Independent3Tests-
Runtime binding
Simulator/Device libraries
Non$instrumented3 Instrumented3application3
application3implementation- implementation-
Monkey$
Sikuli- ?Android?- ?iOS?3-
Runner-
19