Mobile is now facing problems that have been existed in web for many years: fragmentation. In mobile, it is supporting the many phone and tablet models, operating system and versions, and languages. Calabash is an open source technology for functional testing of Android and iOS apps. It combines advanced automation technology with the Cucumber tool. Tests are written in a natural language of the business domain, and the cucumber tool can execute tests on Android and iOS devices or simulators.

1. Introducing Calabash
automated functional testing for mobile native apps
Karl Krukow,
karl@lesspainful.com,
LessPainful
CodeMotion 2012, Rome
1
fredag den 23. marts 12

2. About me
2
fredag den 23. marts 12

3. About me
• PhD, Computer Science, University of Aarhus, 2006.
2
fredag den 23. marts 12

4. About me
• PhD, Computer Science, University of Aarhus, 2006.
• Developer at Trifork for about 6 years mostly Java
enterprise, last two years on iOS.
2
fredag den 23. marts 12

5. About me
• PhD, Computer Science, University of Aarhus, 2006.
• Developer at Trifork for about 6 years mostly Java
enterprise, last two years on iOS.
• Co-owner, iOS responsible at
2
fredag den 23. marts 12

6. About me
• PhD, Computer Science, University of Aarhus, 2006.
• Developer at Trifork for about 6 years mostly Java
enterprise, last two years on iOS.
• Co-owner, iOS responsible at
• Spare-time Hickey & Clojure fan-boy!
2
fredag den 23. marts 12

7. Agenda
3
fredag den 23. marts 12

8. Agenda
• Automated functional testing for native mobile
• Some desirable properties for a functional testing tool
3
fredag den 23. marts 12

9. Agenda
• Automated functional testing for native mobile
• Some desirable properties for a functional testing tool
• Introduce Calabash
• Focus on iOS only
3
fredag den 23. marts 12

10. Agenda
• Automated functional testing for native mobile
• Some desirable properties for a functional testing tool
• Introduce Calabash
• Focus on iOS only
• Live Demo:
• Calabash iOS
• LessPainful: test service and device cloud
3
fredag den 23. marts 12

11. Functional testing
4
fredag den 23. marts 12

12. Functional testing
• Functional and Acceptance tests
• Actual app, as opposed to an isolated component
• Often based on use-cases written in natural language (for domain)
• Visual appearance of screens matter! (Design guidelines, branding,..)
• As realistic an environment as practically possible
4
fredag den 23. marts 12

13. Functional testing
• Functional and Acceptance tests
• Actual app, as opposed to an isolated component
• Often based on use-cases written in natural language (for domain)
• Visual appearance of screens matter! (Design guidelines, branding,..)
• As realistic an environment as practically possible
• For mobile apps, in particular
• Many devices, screens, OS versions, languages
• often a manual process: repetitive, expensive
• regressions: why should I test that again?
4
fredag den 23. marts 12

14. Functional testing
desiderata
5
fredag den 23. marts 12

15. Functional testing
desiderata
• Minimize distance between use cases and actual test
code (DSLs?).
• Expressive and efficient to write.
• Extensible
• High-level, declarative (robustness against “minor” UI
changes).
• Support testing in realistic environments (multiple real
devices, on multiple OS versions, languages).
• Support Continuous integration.
5
fredag den 23. marts 12

16. Cucumber and
Calabash
fredag den 23. marts 12

17. • Cucumber provides
• a framework for writing software specifications
• a tool for executing those specifications
• Specifications are written in a business readable
language that is close to natural language.
• Extremely popular tool for test and specs of
web applications.
• http://cukes.info/
fredag den 23. marts 12

18. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12

19. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12

20. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12

21. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12

22. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12

23. Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,
so I can control access to the application
Scenario: Add test user
When I touch the Add User button
And I fill in text fields as follows:
| field | text |
| Last Name | Knorr |
| Username | knorr |
And I touch "Save"
Then I should be on the Users screen
And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12

24. Step Definitions
• Make the cucumber tests “come alive”
• Written in ordinary programming languages
• Mostly Ruby (but cucumber-jvm: Java, Clojure,...)
Feature Step definitions
fredag den 23. marts 12

25. Step Definitions
• Make the cucumber tests “come alive”
• Written in ordinary programming languages
• Mostly Ruby (but cucumber-jvm: Java, Clojure,...)
Feature Step definitions
Scenario: Add test user
When I touch the Add User button
...
fredag den 23. marts 12

26. Step Definitions
• Make the cucumber tests “come alive”
• Written in ordinary programming languages
• Mostly Ruby (but cucumber-jvm: Java, Clojure,...)
Feature Step definitions
Scenario: Add test user When /^I touch the Add User button$/ do
When I touch the Add User button btn_txt = 'Add user'
... touch("button text:#{btn_txt}")
end
fredag den 23. marts 12

27. Calabash
• One interface: Cucumber, for Android and iOS tests.
• Predefined and custom steps (Ruby + soon: JVM).
• Reuse of tests across platform possible.
• Run on physical devices and simulators.
• Support for hybrid apps (embedded webviews)
• Free, open source with commercial add-ons:
• Run tests in a device cloud using the LessPainful service,
• we can write and run your tests,
• setting up continuous integration, consulting, training, support.
10
fredag den 23. marts 12

28. Architecture iOS
iOS App
--------------------
Calabash library
fredag den 23. marts 12

29. Architecture iOS
features
iOS App
--------------------
Cucumber Calabash library
(ruby api)
Test report
(cucumber)
fredag den 23. marts 12

30. LessPainful
Test Execution in the Cloud
• Execute Calabash tests
concurrently on many
devices, OS’es, languages.
• Authentic: Not jailbroken, iOS
and Android devices, rotation.
• Visual test reports.
• Comparison across models
and operating systems.
• Continuous integration.
fredag den 23. marts 12

31. fredag den 23. marts 12

32. fredag den 23. marts 12

33. fredag den 23. marts 12

34. Calabash iOS:
more detail
15
fredag den 23. marts 12

35. Calabash: more detail
• Very easy to get started for iOS developers/QAs.
• Declarative query language for finding views.
• Based on UISpec, but simplified and extended. (New
Implementation, EPL licensed).
• Touch synthesis, supports (multitouch) gestures.
Extensible. (pan, swipe, pinch, your own)
• Full power of Ruby programming language for test
logic
• Interactive, exploratory development experience.
• Can use device accessibility for identifying views.
16
fredag den 23. marts 12

36. Queries
17
fredag den 23. marts 12

37. Queries
• Queries are like CSS selectors or XPath
• label text:‘Hello‘
• label index:2
• view marked:‘thepane’ label
• view:‘MyClassName’
• label {text LIKE ‘Hel*’}
• webView css:‘#header a.cssclass‘
• webView xpath:’//node()’
17
fredag den 23. marts 12

38. Demo:
- Calabash iOS
- LessPainful Device Cloud
18
fredag den 23. marts 12

39. iOS Comparisons
• Several options available. To my knowledge:
• Calabash
• UIAutomation, Apple
• Zucchini, iOS Testing Framework
• Frank, Pete Hodgson, ThoughtWorks
• UISpec, http://code.google.com/p/uispec/
• FoneMonkey => MonkeyTalk, GorillaLogic
• KIF, Square
• NativeDriver, http://code.google.com/p/nativedriver/
19
fredag den 23. marts 12

40. References
• https://github.com/calabash
• https://github.com/calabash/calabash-ios
• https://github.com/calabash/calabash-ios/wiki
• https://github.com/calabash/calabash-ios-server
• https://github.com/calabash/calabash-android
• http://blog.lesspainful.com/
• https://www.lesspainful.com/
20
fredag den 23. marts 12

41. Questions?
Making app testing less painful...
Please contact us with any questions:
contact@lesspainful.com
karl@lesspainful.com - iOS
jonas@lesspainful.com - Android
http://www.lesspainful.com
fredag den 23. marts 12