Different Android Test Automation Frameworks - What Works You the Best?

Different Android Test
Automation Frameworks –
What works you the best?

21 November 2013
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com

1

Agenda
• Different Ways to Automate Your App Testing
• The Benefits and Characteristics of the Top 5
Android Test Automation Frameworks
• 10 Tips for Professional Mobile App Testing
• Testdroid Update
• Q&A

© Copyrights by Bitbar Technologies Ltd. 2013
All rights reserved.

3

Agenda
• Q&A


4

Different Approaches To Testing


5

Different Ways of Doing Test Automation
Hand written test scripts

Record and Playback

Automatic test exercisers

Fast to create, accurate, not as
sensitive to human-errors as
hand-written tests, tools avail’ty

Fastest & extremely automated,
excellent for smoke testing/quick
testing, availability

Benefits:
Accurate, specific to your testing
needs, plenty of options with
frameworks, tools
Tradeoffs:
Takes a lot of time, ties resources Compelling Recorder+Playback
to write test cases/scripts, error- tools available for only few test
prone (humans)
automation frameworks

Not accurate as real test cases


6

Option #1: Cloud-Based Testing
Users / Consumers

Successful
build

Build
failed

Commit

Repository

Test Cases
Application
QA & Testers

Error reporting

Source code
Developers


7

Option #2: Continuous Integration
Users / Consumers

Successful
build

Build
failed

Commit

Repository

Test Cases
Application
QA & Testers

Error reporting

Source code
Developers


8

Manual

Automation
Large
coverage, quick
ly
completed, Less
money & time
wasted, Exact
results.

Smaller coverage, More
money burnt & time
wasted, Error-prone

9

How to Improve the App Quality
• Automate generic things as much as you can
• During the development your app changes –
the same must apply for testing!
• Carefully select (testing) technology & partner
• Use all 24 hours per day with test automation
• Cloud-based platform is the only solution to
get you quickly covered in the global markets

10

Agenda
• Q&A


11

Android Instrumentation Framework
• InstrumentationTestRunner – the
primary plumbing for running tests
on Android
• Android Instrumentation
Framework is built on top of JUnit
– a standard test framework on for
any Java development
• Mock objects – methods for
creating mock system objects such
as content, service and intent.

12

Family Tree of Test Frameworks
Calabash
Appium

Robotium

Espresso

Android Instrumentation Framework

UI Automator

JUnit


13

Robotium: Basics
• Open source library extending JUnit
with plenty of useful methods for
Android UI testing
• Powerful and robust automatic
black-box test cases for Android
apps
• Supports native, hybrid and mobile
web testing
“It’s like Selenium but for Android”
• Semi-active open source
community with monthly releases

14

Robotium: Code Example
// Public void for the operation
public void testRecorded() throws Exception {
// Wait for the text 'Hello!' to be shown for newbie
if (solo.waitForText("Hello!")) {
// R class ID identifier for 'Sign in' - and click it
solo.clickOnView(solo.findViewById("com.twitter.android.R.id.sign_in"));
// R class ID identifier for entering username
solo.enterText((EditText)
solo.findViewById("com.twitter.android.R.id.login_username"),"username");
// R class ID identifier for entering password
solo.enterText((EditText)
solo.findViewById("com.twitter.android.R.id.login_password"),"password");
// R class ID identifier for clicking log in
solo.clickOnView(solo.findViewById("com.twitter.android.R.id.login_login"));
// Wait until log in is done
solo.waitForActivity("HomeTabActivity");
}
// Activate the text field to compose a tweet
solo.clickOnView(solo.findViewById("com.twitter.android.R.id.menu_compose_tweet"));
// Type the tweet
solo.enterText((EditText) solo.findViewById("com.twitter.android.R.id.edit"), "Testdroid");
// Tweeting!
solo.clickOnView(solo.findViewById("com.twitter.android.R.id.composer_post"));
}


15

Robotium: ExtSolo - Extension Library
• Includes useful methods that have not been
merged to Robotium, for instance:
–
–
–
–
–
–

Automatic scaling of x,y clicks for any resolution
Multi-path drags
Automatic screenshots on test failure
Mock locations
Change device language
Control WiFi connection

• You can download the library at
http://testdroid.com/products/apis-plugins-and-libraries

16

Calabash: Basics
• Behavior driven test framework for native
Android, native iOS and mobile web
• Tests are described in Cucumber and then converted to
Robotium or Frank in run time
• Supports currently about 80 different natural language
commands (controllers). New controllers can be
implemented in Ruby or Java
• If your app uses custom UI Widgets you have to create
controllers for those in order to test your app
• Calabash tools include a command line inspector for
finding right UI element names/ids

17

Calabash: Architecture
• Calabash installs an HTTP server as an
instrumentation package that listens
commands from Calabash server

• Tests are executed on server side
• Each test scenario is described in
Cucumber
• Ruby Client library converts
Cucumber commands to either
Robotium or Frank method calls
• Webview support is implemented
with Javascript injection to the
Webview

Test server

Android device

Step
definitions, Ruby
client library

Calabash HTTP
server


Your app

Features

18

Calabash: Code Example
Feature: Login feature
Scenario: As a valid user I can log into my app
I wait for text "Hello"
Then I press view with id "Sign in"
Then I enter text "username" into "login_username"
Then I enter text "password" into "login_password"
Then I wait for activity "HomeTabActivity"
Then I press view with id "menu_compose_tweet"
Then I enter text "Testdroid" into field with id "edit"
Then I press view with id "composer_post"


19

uiautomator: Basics
• Google’s test framework for testing native Android
apps across device
• Works only on Android API level >=16
• Runs JUnit test cases with special privileges (test
cases can span across different processes)
• No support for web view (only handle to web view
elements as canvas object)


20

uiautomator: Code Example
// Public void for the operation
public void testSignInAndTweet() throws Exception {
// Starting application:
getUiDevice().wakeUp();
// Press Home button to ensure we're on homescreen
getUiDevice().pressHome();
// Select 'Apps' and click button
new UiObject(new UiSelector().description("Apps")).click();
// Select 'Twitter' and click
new UiObject(new UiSelector().text("Twitter")).click();
// Locate and select 'Sign in'
UiSelector signIn = new UiSelector().text("Sign In");
// If button is available, click
UiObject signInButton = new UiObject(signIn);
if (signInButton.exists()) {
signInButton.click();
// Set the username
new UiObject(new
UiSelector().className("android.widget.EditText").instance(0)).setText("username");
new UiObject(new
UiSelector().className("android.widget.EditText").instance(1)).setText("password");
new UiObject(new UiSelector().className("android.widget.Button").
text("Sign In").instance(0)).click();
// Wait Sign in progress window
getUiDevice().waitForWindowUpdate(null, 2000);
// Wait for main window
getUiDevice().waitForWindowUpdate(null, 30000);
}
new UiObject(new UiSelector().description("New tweet")).click();
// Typing text for a tweet
new UiObject(new UiSelector().className("android.widget.LinearLayout").instance(8)).
setText("Awesome #Testdroid!");
// Tweeting!
new UiObject(new UiSelector().text("Tweet")).click();
}


21

Appium: Basics
• Uses Selenium Webdriver (W3C standard) as a
scripting framework
• Supports native Android, native iOS and mobile web:
– Android via uiautomator (API level >=16) and Selendroid
(API level <16)
– iOS via UI Automation
– Mobile web as Selenium driver for Android and iOS

• You can write your Appium scripts on almost any
programming language
(Haskell/Go/Clojure/Java/Ruby)

22

Appium: How it works?
• Appium is an HTTP server that creates and
handles WebDriver sessions
• It starts an Appium server on the device that
is listening commands from the
main Appium process
• It receives json requests from client libraries
over HTTP
• On Android Appium executes these
commands as either uiautomator or
Selendroid commands depending on the API
level of the device
• Tests are driven from a Selenium script on
Appium server

Test server

Android device

UI Automator
controller or
Selendroid driver

Bootstrap.jar

Your app

Selenium script


23

Appium: Test Lifecycle
1. Appium installs Bootstrap.jar as an instrumentation package
which contains an uiautomator server and a tcp server
2. The commands from a Selenium script are executed on the
main Appium server that relays the commands over TCP/IP
to the tcp server running on the device
3. The UI Automation server converts the Selenium commands
to uiautomator commands on the fly
4. If the device has API level <=17 Appium installs a Selendroid
server that converts the Selenium commands to Android
Instrumentation (JUnit) commands
5. All test verification and result processing is happening on
server side in the main Appium Server

24

Appium: Code Example
# wait for hello
sleep(3)
textFields = driver.find_elements_by_tag_name('textField')
assertEqual(textFields[0].get_attribute("value"), "Hello")
# click sign-in button
driver.find_elements_by_name('Sign in')[0].click()
# find the text fields again, and enter username and password
textFields = driver.find_elements_by_tag_name('textField')
textFields[0].send_keys("twitter_username")
textFields[1].send_keys("passw0rd")
# click the Login button (the first button in the view)
driver.find_elements_by_tag_name('button')[0].click()
# sleep
sleep(3)
# click the first button with name "Compose"
driver.find_elements_by_name('Compose')[0].click()
# type in the tweet message
driver.find_elements_by_tag_name('textField')[0].send_keys(”#Testdroid is awesome!")
# press the Send button
driver.find_elements_by_name('Send')[0].click()
# exit
driver.quit()


25

Espresso: Basics
•
•
•
•
•

The latest Android test automation framework from Google
A custom Instrumentation Testrunner with special privileges
Works on API levels 8 (Froyo), 10 (Gingerbread), 15-19 (IJK)
Thin layer on top of Android Instrumentation Framework
Uses the Hamcrest matcher library
https://github.com/hamcrest


26

Espresso: Basics 2
• Easy API for extending the framework:
– You can write new matchers: onView(myCustomMatcher<View>)
– You can write new actions: perform(myCustomAction)
– You can write new checks: check(myCustomAssertion)

• Reliable: Synchronizes with the UI thread
• It’s fast because there is no need for any sleeps (tests run on
same millisecond when the app becomes idle)
• No support for web views
• https://code.google.com/p/android-test-kit/


27

Espresso: Code Example
public void testEspresso() {
// Check if view with the text 'Hello.' is shown
onView(withText("Hello.")).check(matches(isDisplayed()));
// R class ID identifier for 'Sign in' - and click it
onView(withId(getInstrumentation().getTargetContext().getResources()
.getIdentifier("com.twitter.android:id/sign_in", null, null))).perform(click());
// R class ID identifier for entering username
.getIdentifier("com.twitter.android:id/login_username", null, null))).perform((typeText("username")));
// R class ID identifier for entering password
.getIdentifier("com.twitter.android:id/login_password", null, null))).perform((typeText("password")));
// R class ID identifier for clicking log in
.getIdentifier("com.twitter.android:id/login_login", null, null))).perform(click());
// Activate the text field to compose a tweet
.getIdentifier("com.twitter.android:id/menu_compose_tweet", null, null))).perform(click());
// Type the tweet
.getIdentifier("com.twitter.android:id/edit", null, null))).perform((typeText(”#Testdroid")));
// Tweeting!
.getIdentifier("com.twitter.android:id/composer_post", null, null))).perform(click());
}


28

Summary – Comparison Matrix
Robotium

uiautomator

Espresso

Appium

Calabash

Android

Yes

Yes

Yes

Yes

Yes

iOS

No

No

No

Yes

Yes

Mobile web

Yes

No

Yes

Yes

(Android)

Limited to x.y
clicks

(Android & iOS)

(Android)

Java

Java

Java

Almost any

Ruby

Testdroid
Recorder

UI Automator
viewer

Hierarchy
Viewer

Appium.app

CLI

Supported
API levels

All

16 =>

8, 10, 15-19

All

All

Community

Contributors

Google

Google

Active

Pretty quiet

Scripting
Language
Test creation
tools


29

Agenda
• Q&A


30

Tip #1: Test early, Test often


32

Tip #2: Plan What to Automate
Time

MANUAL

Infrastructure
Tools

AUTOMATED

People

Training


33

Tip #3: Use only Real Devices


34

Tip #4: Use Atomic Test Units

Test iteration #1

Test iteration #2


35

Tip #5: Create Hermetic Tests


36

Tip #6: Use All Possible Devices

Testdroid Cloud’s 250+ Android devices
= 93-95% global Android volumes!

37

Tip #7: Separate Apps and Tests

Your App


38

Tip #8: Output Everything to Logs


39

Tip #9: End-to-End Testing


40

Tip #10: Integrate Dev & Testing
Repository

Successful
build

Commit

Users / Consumers

Build
failed

Test Cases
Application
QA & Testers

Error reporting

Source code
Developers


41

Agenda
• Q&A


42

Testdroid Products
Complete Solution for Mobile Apps/Games Testing


43

Testdroid
– Because it is important to know what app testing costs!


44

Testdroid Blog and Webinars
– Because it is important to how to automate your testing!


45

Agenda
• Q&A


46


47

Different Android Test Automation Frameworks - What Works You the Best?

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (19)

Similar to Different Android Test Automation Frameworks - What Works You the Best?

Similar to Different Android Test Automation Frameworks - What Works You the Best? (20)

More from Bitbar

More from Bitbar (9)

Recently uploaded

Recently uploaded (20)

Different Android Test Automation Frameworks - What Works You the Best?