Rajdeep Varma discusses parallel_calabash, a tool they created to run mobile test automation in parallel. It addresses long test run times by distributing Cucumber feature files across multiple device instances and processes. This allows taking advantage of modern multi-core machines and significantly increases testing velocity while maintaining test coverage. It can run tests on multiple real or emulator devices simultaneously and generates separate JUnit-style reports for each device. With parallel_calabash, they achieved a 70% increase in testing speed while gaining the ability to test on multiple device configurations in parallel.

2. About Myself
• Rajdeep Varma
• 5 years in Quality Engineering. I work at
ThoughtWorks
• Passionate about test automation and agile
practices
• Tested a number of web apps and recently
started testing mobile apps

3. I will talk about
• Our Problem:
A. Long running automation
B. Running Automation on Multiple devices
• Our Solution: parallel_calabash

4. Typical CI Pipeline Of Mobile Project
Git Push
Compile
+
Unit Test
Build App UI Tests Distribute
“Fast Feedback”

6. Stats
• 5 git repo
• Avg. 30 commits a day
• Approx. 100 Automated UI tests.
• “GitPush” TO “TestedOK” > 1 Hrs

7. With slow tests
Git Push
UI Tests
Deploy to
next stage
Umm…slow tests—Chai Break—damn! test failed :(
Fixed!
<pain>its so slow<pain/>— Pray —

8. LOL

9. :(

10. Patience Is Not Always A Virtue

11. Choose One
a) Lots of automated tests for good coverage
b) Faster automation run
c) BOTH!

12. How?
http://sharpreflections.com/wp-content/uploads/2012/06/multi_core_cpu.png

13. Our tool-set
• MobileWeb: Cucumber + waitr-webdriver + parallel_tests
• AndroidApp: Calabash

14. Options
• Multiple parallel builds in CI
• Proprietary cloud based tools
• Create a tool that matches our need
http://images.clipartpanda.com/choice-clipart-fork-choice-hi.png

15. parallel_calabash

16. Demo

19. Single Device Parallel
Time
CPUUsage

20. How it works?
<feature_grouper>
1_digit_addition.feature
2_digit_addition.feature
3_digit_addition.feature
4_digit_addition.feature
5_digit_addition.feature
6_digit_addition.feature
7_digit_addition.feature
8_digit_addition.feature
Detect
Devices
Feature
Grouper
1_digit_addition
2_digit_addition
3_digit_addition
4_digit_addition
5_digit_addition
6_digit_addition
7_digit_addition
8_digit_addition

21. How it works?
<process spawner>
1_digit_addition
2_digit_addition
3_digit_addition
4_digit_addition
5_digit_addition
6_digit_addition
7_digit_addition
8_digit_addition
ENV[ADB_DEVICE_ARG] = emulator-5554
ENV[TEST_PROCESS_NUMBER] = 1
Process
Spawner
ENV[ADB_DEVICE_ARG] = emulator-5556
ENV[TEST_PROCESS_NUMBER] = 2
ENV[ADB_DEVICE_ARG] = emulator-5558
ENV[TEST_PROCESS_NUMBER] = 3
ENV[ADB_DEVICE_ARG] = emulator-5560
ENV[TEST_PROCESS_NUMBER] = 4
S
U
M
M
A
R
I
Z
E
R
E
S
U
L
T
stdout

22. Reporting
config/cucumber.yml
default: --format html --out my_report_<%=ENV['DEVICE_INFO']%>.html
Reports from 3 devices:
my_report_GT_N7100.html
my_report_Samsung_Galaxy_Note_2___4_3___API_18___720x1280_1.html
my_report_Samsung_Galaxy_Note_2___4_3___API_18___720x1280_2.html

23. Optimization in test
distribution
• Issue with distribution based on cucumber
“featue” files
• Distribution based on “Scenario” instead of
features
• --group-by-scenarios

24. Advantages of fast tests
• Fast Feedback
• Faster development cycle for BDD environment
• Modern machines have multiple cores and
parallelization can make Maximum use of
Hardware
• Faster tests makes developers & Testers happy

25. I will talk about
• Our Problem:
A. Long running automation
B. Running Automation on Multiple devices
• Our Solution: parallel_calabash

26. Device Fragmentation
parallel_calabash -a <path/to/apk> features --concurrent

27. what did we achieve?
• 70% increase in testing velocity.
• Scalability
• Reporting
• Process specific action using
ENV[TEST_PROCESS_NUMBER]

29. Choose One
a) Lots of automated tests for good coverage
b) Faster automation run
c) BOTH!

30. TIME

31. https://github.com/rajdeepv/parallel_calabash
https://rubygems.org/gems/parallel_calabash
parallel_calabash

32. References:
• https://github.com/grosser/parallel_tests
• http://martinfowler.com/
• http://continuousdelivery.com/
• http://watirmelon.com/

33. ?