iOS Automation rapidly changed due to the evolution of Apple automation frameworks and Calabash community support. One year ago, we parallelised our iOS testing with multiple desktops – a large improvement in performance. Here is an account of the subsequent changes, including the removal of multiple desktops and the adoption of Apple’s new XCTest framework. The automation community (Appium, Calabash etc.) faced a new challenge: how could we create iOS Automation Infrastructure with the new XCTest Framework? I adopted Facebook solution (FBSimCtl and WebAgentDriver) into the multi-simulators approach. In this talk, I will discuss the full evolution path to Facebash Approach based on multi-simulators, Calabash server and Facebook Web Driver Agent.

1. iOS Parallel Automation:
Run faster than fast!
Viktar Karanevich

2. Content:
I. History of iOS Automation Frameworks (5 min)
II. Path to Current Approach at Badoo (10 min)
III. Summary (5 min)
IV. Movie: Test Run (3 min)

3. self.introduce =>
{
name: "Viktar Karanevich",
title: ["SSTAE", "SSDET", "SQAAE"],
languages: ["Java", "Ruby"],
experience: "iOS Automation 7+ years"
}

4. The original, largest and leading dating network:
330M+ registered users
800+ automated tests on iOS platform
50M+ downloads on iOS

5. Steve Jobs
Steve
Wozniak
I. History

6. Jun 2010
UI Automation
Apple

7. Jun 2010
UI Automation
Sep 2014
UI Automation
xcrun simctl
Apple
"xcrun simctl" is introduced in xCode 6
It is command line utility to control the Simulator
(create, delete, erase, install, uninstall, ...).

8. Jun 2010
UI Automation
Sep 2015
UI Automation
xcrun simctl
UI Testing
Apple
UI Testing is introduced as replacement
for UIAutomation in xCode 7
Sep 2014
UI Automation
xcrun simctl

9. Jun 2010
UI Automation
Sep 2016
UI Testing
xcrun simctl
Apple
Sep 2015
UI Automation
xcrun simctl
UI Testing
Sep 2014
UI Automation
xcrun simctl
UI Automation removed in xCode 8

11. Jun 2010
UI Automation
Sep 2014
xcrun simctl
Sep 2016
UI Testing
Sep 2016
UI Testing
Apple
Calabash
Nov 2009
UI Spec

12. Calabash
Nov 2009
UI Spec
Feb 2011
Frank

13. Calabash
Nov 2009
UI Spec
Feb 2011
Frank
Jan 2012
Calabash
Step Definitions
Ruby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQuery
Touch Actions

14. Calabash
Apr 2013
Calabash +
UIAutomation
Link: https://github.com/calabash/run_loop
Step Definitions
Ruby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQuery
Run loop:
UIAutomation

15. Calabash
Sep 2016
Calabash +
DeviceAgent
Link: https://github.com/calabash/run_loop/
tree/develop/lib/run_loop/device_agent
Developer host Simulator
Step Definitions
Ruby Client Library
XCTest Runner
Features
Calabash server
Your app
UIQuery
Device Agent

16. Calabash
Nov 2009
UI Spec
Feb 2011
Frank
Jan 2012
Calabash
iOSAutomation Frameworks
Apr 2013
Calabash +
UIAutomation
Sep 2016
Calabash +
DeviceAgent
Jun 2010
UI Automation
Sep 2016
UI Testing
xcrun simctl
Sep 2015
UI Automation
xcrun simctl
UI Testing
Sep 2014
UI Automation
xcrun simctl
Apple
3 years 1 year

17. Calabash
Nov 2009
UI Spec
Feb 2011
Frank
Jan 2012
Calabash
iOSAutomation Frameworks
Apr 2013
Calabash +
UIAutomation
Sep 2016
Calabash +
DeviceAgent
Jun 2010
UI Automation
Sep 2016
UI Testing
xcrun simctl
Sep 2015
UI Automation
xcrun simctl
UI Testing
Sep 2014
UI Automation
xcrun simctl
Apple
3 years 1 year

18. II. iOS Automation at Badoo
Approach
~80
flaky tests
80
data
fail tests

19. II. Before Parallel Run
Test Suite Run
Nightly Cucumber Report
Nightmare!
15h
total time
1
simulator
80
data
~500
tests

20. II. Before Parallel Run
Test Suite Run
Nightly Cucumber Report
Nightmare!
80
data
fail tests
1simulator
Key Issue: How to speed up Test Run?
15h
total time
~500
tests
1
simulator

21. Let's Parallel!

22. Let's Parallel!
Blocker Issue:
- Calabash does not support Parallel run on 1 desktop

23. Let's Parallel
with multi-desktops!

24. https://techblog.badoo.com
Tim's talk at Badoo: "Parallel Calabash Testing on iOS"

25. II. Test Result: multi-desktops
~4 hours regression suite run time
5 Desktops

26. II. Test Result: multi-desktops
Mac PRO (24 cores)
~2 hours regression suite run time
10 Desktops

28. X Desktops: host limit reached

29. II. Clustering Hosts

30. II. Clustering Hosts
- Parallel run on multi (virtual) hosts
- Mac Pro and hosts: test suite - ~1h 30 min

31. II. Clustering Hosts
Issues:
- multi-desktops start up is Flaky
- simulators can get stuck

32. II. Clustering Hosts
Is it possible to automate multi-simulators on one desktop?

33. III. Facebook: Web Driver Agent

34. Developer host Simulator
Selenium Driver
WDA Runner
Your app
FB UIA WDA
Key Change: Reimplemented UIAutomation dependency
Link: https://github.com/facebook/WebDriverAgent

35. Profit:
- 7 simulators on single desktop
- More stable! more efficient!

36. III. Full Migration to FB UIA
WDA
Issues:
- all tests have calabash code

37. III. Full Migration to FB UIA
WDA
Issues:
- all tests have calabash code
- custom app backdoors

38. III. Full Migration to FB UIA
WDA
Issues:
- all tests have calabash code
- custom app backdoors
- workarounds and other

39. III. Full Migration to FB UIA
WDA
Full Migration Costs: ~x hours
Is it possible to do Fast Migration?

40. III. Calabash + Facebook
Fast migration: Facebash!
Step Definitions
Ruby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQuery
Run loop:
UIAutomation

41. III. Calabash + Facebook
Fast migration: Facebash!
Step Definitions
Ruby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQuery
Run loop:
UIAutomation
Developer host Simulator
Step Definitions
Ruby Client Library
WDA Runner
Features
Calabash server
Your app
UIQuery
FB UIA WDA

42. IV. iOS 10 Update:
from UIAutomation WDAto XCTest WDA
Link: https://github.com/facebook/WebDriverAgent
Developer host Simulator
Selenium Driver
Your app
FB UIA WDA
WDA Runner

43. IV. iOS 10 Update:
from UIAutomation WDAto XCTest WDA
Link: https://github.com/facebook/WebDriverAgent
Developer host Simulator
Selenium Driver
Your app
FB UIA WDA
Developer host Simulator
Your app
FB XCTest WDA
WDA Runner
Selenium Driver
FB simctl

44. IV. iOS Hybrid 2.0
iOS 10 update
Developer host Simulator
Step Definitions
Ruby Client Library
WDA Runner
Features
Calabash server
Your app
UIQuery
FB UIA WDA

45. IV. iOS Hybrid 2.0
iOS 10 update
Developer host Simulator
Step Definitions
Ruby Client Library
WDA Runner
Features
Calabash server
Your app
UIQuery
FB UIA WDA
Developer host Simulator
Step Definitions
Ruby Client Library
FB simctl
Features
Calabash server
Your app
UIQuery
FB XCTest WDA

46. V. Summary: multi-desktops vs multi-simulators
Test Suite: 500 tests
Nightly Cucumber Report
Happy morning!
80
data
Mac PRO
Test Suite Run Time
0 min
225 min
450 min
675 min
900 min
1 sim 10 sims 16 sims
75 min
120 min
900 min
Multi
Desktops
Multi
Simulators

47. Developer host Simulator
Step Definitions
Ruby Client Library
FB simctl
Features
Calabash server
Your app
UIQuery
FB XCTest WDA
"Face"book + Cala"bash" = Facebash
V. Summary: current approach

48. - easy to set up Parallel Run on 1 Desktop
- keep using of backdoors
- full access to internal AUT state
- cross iOS app automation
- iOS simulator control with FBSimCtl
- FB WDA supports Selenium protocol
- FB WDA independent of Calabash/Appium
Facebook
Calabash
Calabash
Facebook
Facebook
Facebook
Facebook
V. Summary: PROs

49. - support framework parts: calabash server, FB WDA, fbsimctl, ...
- "poor" documentation
- using of mixed locator strategies
V. Summary: PROs and CONs

51. - Parallel Run in Headless Mode
- Simulator Start/Shutdown/Reset optimization
- Cold/Hot start iOS app with relogin
- record video per the each test in Parallel Run
V. TODO

52. Thank You!
v.karanevich@corp.badoo.com
Questions?