Testdroid:

Tips & Tricks with
Calabash for
Mobile App Testing
13 March 2014
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com

Agenda
• Prerequisites for Mobile App Testing
• Calabash Introduction and How It Compares to
Other Test Automation Frameworks
• Inside the Calabash – Building Those Scripts
• Tips & Tricks with Calabash
• Demonstration
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
2

Agenda
• Tips and Tricks with Calabash
• Demonstration
• Q&A
3

Why Apps Fail?
4

Prerequisites for Mobile App Testing
• Testing environment as authentic as possible
– Real devices <-> emulators/simulators
• Test coverage needs to be adequate
– Software is not identical in all Android/iOS devices
– Hardware is even more fragmented landscape
• Surrounding testing conditions (e.g. network)
• Try not to build in repetitive, manual processes
• Test automation framework that works for you
5

What Framework Works You The Best?
• High-level & declarative test automation
frameworks work well when app is under dev
• Easy to integrate with continuous integration
• Distance between use case and actual test
• What do you build: Native, Hybrid or Web app?
• The technical competence in your organization
• Desired outcome from test automation
6

What Improves the App Quality?
• Automate generic things as much as you can
• During the development your app changes –
the same must apply for testing!
• Carefully select (testing) technology & partner
• Use all 24 hours per day with test automation
• Cloud-based platform is the only solution to
get you quickly covered in the global markets
7

Why Real Devices are Must-to-Have?
• Emulators/simulators won’t help you to test…
– User Experience and Usability
– Hardware
– Software
– Infrastructure
0 % = the percentage of your app users
that use emulator to run your app!
8

Basics: Different Ways of Doing Test Automation
Automatic test exercisersRecord and PlaybackHand written test scripts
Benefits:
Accurate, specific to your testing
needs, plenty of options with
frameworks, tools
Fast to create, accurate, not as
sensitive to human-errors as
hand-written tests, tools avail’ty
Fastest & extremely automated,
excellent for smoke testing/quick
testing, availability
Tradeoffs:
Takes a lot of time, ties resources
to write test cases/scripts, error-
prone (humans)
Compelling Recorder+Playback
tools available for only few test
automation frameworks
Not accurate as real test cases
9

Agenda
• Calabash Introduction and How It Compares
to Other Test Automation Frameworks
• Demonstration
• Q&A
10

Introduction to Calabash
• Behavior driven test framework for native Android &
iOS, and for hybrid web apps (embedded webviews)
• Tests are described in Cucumber and then converted to
Robotium or “similar-to-Frank“ in run time
• Supports currently about 80 different natural language
commands (controllers). New controllers can be
implemented in Ruby or Java
• If your app uses custom UI Widgets you have to create
controllers for those in order to test your app
• Calabash tools include a command line inspector for
finding right UI element names/ids
11

Android Example: The Family Tree
of Test Automation Frameworks
JUnit
Android Instrumentation Framework
Robotium Espresso
UI Automator
Calabash
Appium
12

Why Calabash is A Good Choice?
• Cross-platform test automation framework
• Human-readable syntax/tests
• Not much changes needed for tests when the
mobile app changes (e.g. UI components)
• Pretty easy to integrate with any CI
• Great choice for teams that have limited
resources for development and/or testing
13

Calabash-iOS: Architecture
• Calabash installs an HTTP server as an
instrumentation package that listens
commands from Calabash server
• Tests are executed on server side
• Each test scenario is described in
Cucumber
• Ruby Client library converts Cucumber
commands to ”Frank” method calls
• Webview support is implemented with
Javascript injection to the Webview
iOS device
Your app
Calabash HTTP
server
Test server
Step
definitions, Ruby
client library
Features
14

Calabash-Android: Architecture
• Calabash uses Android instrumentation
test server (a separate app)
• Based on
ActivityInstrumentationTestCase2
• Tests are executed on server side
• Predefined and custom steps
• Ruby Client library converts Cucumber
commands to Robotium method calls
• Webview support is implemented with
Javascript injection to the Webview
Android device
Your app
Instrumentation
Test Server
Test server
Step
definitions, Ruby
client library
Features
15

Calabash: Code Example
Feature: Login feature
Scenario: As a valid user I can log into my app
I wait for text "Hello"
Then I press view with id "Sign in"
Then I enter text "username" into "login_username"
Then I enter text "password" into "login_password"
Then I wait for activity "HomeTabActivity"
Then I press view with id "menu_compose_tweet"
Then I enter text "Testdroid" into field with id "edit"
Then I press view with id "composer_post"
16

How Calabash Compares to Others?
Calabash Robotium uiautomator Espresso Appium
Android Yes Yes Yes Yes Yes
iOS Yes No No No Yes
Mobile web Hybrid
(webviews)
Yes
(Android)
Limited to x.y
clicks
No Yes
(Android & iOS)
Scripting
Language
Ruby Java Java Java Almost any
Test creation
tools
CLI Testdroid
Recorder
UI Automator
viewer
Hierarchy
Viewer
Appium.app
Supported
API levels
All All 16 => 8, 10, 15-19 All
Community Active Contributors Google Google Active
17

Example #1: Calabash with Cloud
Developers
Source code
Repository
Build
failed
Successful
build
QA & Testers
Error reporting
Test Cases
Application
Users / Consumers
Commit
18

Example #2: Calabash with CI
Developers
Source code
Repository
Build
failed
Successful
build
QA & Testers
Error reporting
Test Cases
Application
Users / Consumers
Commit
19

Agenda
• Demonstration
• Q&A
20

Step Definitions
• Fast and easy to modify definitions and features
• Predefined and Custom steps
– Predefined steps offer the way to start instantly
– Custom steps provide more options & can be easily
tailored for your app
• Works for both, Android and iOS, when added:
– calabash.framework
– ActivityInstrumentationTestCase2
• Custom steps extremely easy to modify & create
21

Test Script – called Features
22
Feature: Login feature
Scenario: As a valid user I can log into my app
I wait for text "Hello"
Then I press view with id "Sign in"
Then I enter text "username" into "login_username"
Then I enter text "password" into "login_password"
Then I wait for activity "HomeTabActivity"
Then I enter text "Testdroid" into field with id "edit"
Then I press view with id "composer_post"

Example: Predefined Steps
23
Given /Î press the "([^"]*)" button$/ do |buttonText|
performAction('press_button_with_text', buttonText)
end
Then /Î press button number (d+)$/ do |buttonNumber|
performAction('press_button_number', buttonNumber)
end
Then /Î press image button number (d+)$/ do |buttonNumber|
performAction('press_image_button_number', buttonNumber)
end
Then /Î press view with id "([^"]*)"$/ do |view_id|
performAction('click_on_view_by_id',view_id)
end

From Definition to Feature
24
Then /^I press view with id "([^"]*)"$/ do |view_id|
performAction('click_on_view_by_id',view_id)
end
Definition:
Feature:

Agenda
• Building the Calabash Scripts
• Demonstration
• Q&A
25

#1: Use Only Real Devices
26

#2: Calabash is Perfect for
“Test Early, Test Often” Approach
27

#3: Integrate Dev & Testing -> CI
28
Commit
Developers
Source code
Repository
Build
failed
Successful
build
QA & Testers
Error reporting
Test Cases
Application
Users / Consumers

Agenda
• Demonstration
• Q&A
29

Agenda
• Demonstration
• Q&A
30

31

Testdroid:

More Related Content

What's hot

Similar to Testdroid:

Testdroid: