More Related Content
Similar to Testdroid: (20)
Testdroid:
- 1. Tips & Tricks with
Calabash for
Mobile App Testing
13 March 2014
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com
- 2. Agenda
• Prerequisites for Mobile App Testing
• Calabash Introduction and How It Compares to
Other Test Automation Frameworks
• Inside the Calabash – Building Those Scripts
• Tips & Tricks with Calabash
• Demonstration
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
2
- 3. Agenda
• Prerequisites for Mobile App Testing
• Calabash Introduction and How It Compares to
Other Test Automation Frameworks
• Inside the Calabash – Building Those Scripts
• Tips and Tricks with Calabash
• Demonstration
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
3
- 4. Why Apps Fail?
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
4
- 5. Prerequisites for Mobile App Testing
• Testing environment as authentic as possible
– Real devices <-> emulators/simulators
• Test coverage needs to be adequate
– Software is not identical in all Android/iOS devices
– Hardware is even more fragmented landscape
• Surrounding testing conditions (e.g. network)
• Try not to build in repetitive, manual processes
• Test automation framework that works for you
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
5
- 6. What Framework Works You The Best?
• High-level & declarative test automation
frameworks work well when app is under dev
• Easy to integrate with continuous integration
• Distance between use case and actual test
• What do you build: Native, Hybrid or Web app?
• The technical competence in your organization
• Desired outcome from test automation
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
6
- 7. What Improves the App Quality?
• Automate generic things as much as you can
• During the development your app changes –
the same must apply for testing!
• Carefully select (testing) technology & partner
• Use all 24 hours per day with test automation
• Cloud-based platform is the only solution to
get you quickly covered in the global markets
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
7
- 8. Why Real Devices are Must-to-Have?
• Emulators/simulators won’t help you to test…
– User Experience and Usability
– Hardware
– Software
– Infrastructure
0 % = the percentage of your app users
that use emulator to run your app!
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
8
- 9. Basics: Different Ways of Doing Test Automation
Automatic test exercisersRecord and PlaybackHand written test scripts
Benefits:
Accurate, specific to your testing
needs, plenty of options with
frameworks, tools
Fast to create, accurate, not as
sensitive to human-errors as
hand-written tests, tools avail’ty
Fastest & extremely automated,
excellent for smoke testing/quick
testing, availability
Tradeoffs:
Takes a lot of time, ties resources
to write test cases/scripts, error-
prone (humans)
Compelling Recorder+Playback
tools available for only few test
automation frameworks
Not accurate as real test cases
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
9
- 10. Agenda
• Prerequisites for Mobile App Testing
• Calabash Introduction and How It Compares
to Other Test Automation Frameworks
• Inside the Calabash – Building Those Scripts
• Tips and Tricks with Calabash
• Demonstration
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
10
- 11. Introduction to Calabash
• Behavior driven test framework for native Android &
iOS, and for hybrid web apps (embedded webviews)
• Tests are described in Cucumber and then converted to
Robotium or “similar-to-Frank“ in run time
• Supports currently about 80 different natural language
commands (controllers). New controllers can be
implemented in Ruby or Java
• If your app uses custom UI Widgets you have to create
controllers for those in order to test your app
• Calabash tools include a command line inspector for
finding right UI element names/ids
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
11
- 12. Android Example: The Family Tree
of Test Automation Frameworks
JUnit
Android Instrumentation Framework
Robotium Espresso
UI Automator
Calabash
Appium
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
12
- 13. Why Calabash is A Good Choice?
• Cross-platform test automation framework
• Human-readable syntax/tests
• Not much changes needed for tests when the
mobile app changes (e.g. UI components)
• Pretty easy to integrate with any CI
• Great choice for teams that have limited
resources for development and/or testing
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
13
- 14. Calabash-iOS: Architecture
• Calabash installs an HTTP server as an
instrumentation package that listens
commands from Calabash server
• Tests are executed on server side
• Each test scenario is described in
Cucumber
• Ruby Client library converts Cucumber
commands to ”Frank” method calls
• Webview support is implemented with
Javascript injection to the Webview
iOS device
Your app
Calabash HTTP
server
Test server
Step
definitions, Ruby
client library
Features
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
14
- 15. Calabash-Android: Architecture
• Calabash uses Android instrumentation
test server (a separate app)
• Based on
ActivityInstrumentationTestCase2
• Tests are executed on server side
• Predefined and custom steps
• Ruby Client library converts Cucumber
commands to Robotium method calls
• Webview support is implemented with
Javascript injection to the Webview
Android device
Your app
Instrumentation
Test Server
Test server
Step
definitions, Ruby
client library
Features
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
15
- 16. Calabash: Code Example
Feature: Login feature
Scenario: As a valid user I can log into my app
I wait for text "Hello"
Then I press view with id "Sign in"
Then I enter text "username" into "login_username"
Then I enter text "password" into "login_password"
Then I wait for activity "HomeTabActivity"
Then I press view with id "menu_compose_tweet"
Then I enter text "Testdroid" into field with id "edit"
Then I press view with id "composer_post"
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
16
- 17. How Calabash Compares to Others?
Calabash Robotium uiautomator Espresso Appium
Android Yes Yes Yes Yes Yes
iOS Yes No No No Yes
Mobile web Hybrid
(webviews)
Yes
(Android)
Limited to x.y
clicks
No Yes
(Android & iOS)
Scripting
Language
Ruby Java Java Java Almost any
Test creation
tools
CLI Testdroid
Recorder
UI Automator
viewer
Hierarchy
Viewer
Appium.app
Supported
API levels
All All 16 => 8, 10, 15-19 All
Community Active Contributors Google Google Active
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
17
- 18. Example #1: Calabash with Cloud
Developers
Source code
Repository
Build
failed
Successful
build
QA & Testers
Error reporting
Test Cases
Application
Users / Consumers
Commit
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
18
- 19. Example #2: Calabash with CI
Developers
Source code
Repository
Build
failed
Successful
build
QA & Testers
Error reporting
Test Cases
Application
Users / Consumers
Commit
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
19
- 20. Agenda
• Prerequisites for Mobile App Testing
• Calabash Introduction and How It Compares to
Other Test Automation Frameworks
• Inside the Calabash – Building Those Scripts
• Tips and Tricks with Calabash
• Demonstration
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
20
- 21. Step Definitions
• Fast and easy to modify definitions and features
• Predefined and Custom steps
– Predefined steps offer the way to start instantly
– Custom steps provide more options & can be easily
tailored for your app
• Works for both, Android and iOS, when added:
– calabash.framework
– ActivityInstrumentationTestCase2
• Custom steps extremely easy to modify & create
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
21
- 22. Test Script – called Features
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
22
Feature: Login feature
Scenario: As a valid user I can log into my app
I wait for text "Hello"
Then I press view with id "Sign in"
Then I enter text "username" into "login_username"
Then I enter text "password" into "login_password"
Then I wait for activity "HomeTabActivity"
Then I press view with id "menu_compose_tweet"
Then I enter text "Testdroid" into field with id "edit"
Then I press view with id "composer_post"
- 23. Example: Predefined Steps
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
23
Given /^I press the "([^"]*)" button$/ do |buttonText|
performAction('press_button_with_text', buttonText)
end
Then /^I press button number (d+)$/ do |buttonNumber|
performAction('press_button_number', buttonNumber)
end
Then /^I press image button number (d+)$/ do |buttonNumber|
performAction('press_image_button_number', buttonNumber)
end
Then /^I press view with id "([^"]*)"$/ do |view_id|
performAction('click_on_view_by_id',view_id)
end
- 24. From Definition to Feature
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
24
Then /^I press view with id "([^"]*)"$/ do |view_id|
performAction('click_on_view_by_id',view_id)
end
Then I press view with id "menu_compose_tweet"
Definition:
Feature:
- 25. Agenda
• Prerequisites for Mobile App Testing
• Calabash Introduction and How It Compares to
Other Test Automation Frameworks
• Building the Calabash Scripts
• Tips and Tricks with Calabash
• Demonstration
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
25
- 26. #1: Use Only Real Devices
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
26
- 27. #2: Calabash is Perfect for
“Test Early, Test Often” Approach
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
27
- 28. #3: Integrate Dev & Testing -> CI
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
28
Commit
Developers
Source code
Repository
Build
failed
Successful
build
QA & Testers
Error reporting
Test Cases
Application
Users / Consumers
- 29. Agenda
• Prerequisites for Mobile App Testing
• Calabash Introduction and How It Compares to
Other Test Automation Frameworks
• Inside the Calabash – Building Those Scripts
• Tips and Tricks with Calabash
• Demonstration
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
29
- 30. Agenda
• Prerequisites for Mobile App Testing
• Calabash Introduction and How It Compares to
Other Test Automation Frameworks
• Inside the Calabash – Building Those Scripts
• Tips and Tricks with Calabash
• Demonstration
• Q&A
© Copyrights by Bitbar Technologies Ltd. 2014
All rights reserved.
30