The document summarizes the shifting landscape of mobile test automation over the past 5 years. It discusses the major open source and proprietary frameworks for iOS and Android testing, including Appium, Espresso, XCUITest, EarlGrey, Detox, and others. It provides an overview of each framework's capabilities and adoption. It also envisions where mobile automation may be headed in the next 5 years, such as supporting new form factors, platforms, and types of testing beyond functional testing.

1. The Shifting Landscape of Mobile Automation
Jonathan Lipps • Founding Principal • Cloud Grey
@AppiumDevs • @cloudgrey_io • @jlipps
Applitools Webinar · The Internet
February 23, 2018

2. Founding Principal
Project Lead & Architect
Jonathan Lipps • Founding Principal • Cloud Grey
@AppiumDevs • @cloudgrey_io • @jlipps

3. 5 years ago

4. Android Instrumentation

5. Open Source Proprietary
Appium
Android Instrumentation*
Android UiAutomator*
Robotium
Frank
Selendroid
ios-driver
KIF
Calabash
Apple Instruments +
UIAutomation

6. Today

7. XCUITest
(iOS)
EarlGrey
(iOS)
Detox
(iOS + Android)
Espresso
(Android)
Appium
(Many)
Xamarin.UITest
(iOS + Android)
🛠 🛠
WebDriverAgent
(iOS)
🛠🛠🛠

8. Open Source (Owner) Proprietary
Appium (JS Foundation)
Espresso* (Google)
EarlGrey (Google)
Detox (Wix)
Xamarin.UITest (Microsoft)
WebDriverAgent (Facebook)
XCUITest (Apple)

9. Framework Platforms Modes Languages Protocol
Requires
Source?
Underlying
Tech /
Vendor
IDE/SDK
integration
Service
Providers
Dev
Appium
iOS, Android,
macOS,
Windows
Native, Hybrid,
Web
Java, Python, Ruby,
C#, JavaScript,
Objective-C, Swift,
PHP
WebDriver No
WDA,
UiAutomator2,
Espresso,
WinAppDriver,
appium-for-mac
N/A
Sauce Labs,
TestObject, AWS,
BrowserStack, BitBar,
Xamarin, AppCenter,
etc…
Full OSS
Espresso Android Native, Hybrid Java, Kotlin Non-standard Yes Google Android Studio
FireBase, BitRise,
TestObject, BitBar,
AWS
Semi OSS
XCUITest iOS Native
Objective-C,
Swift
Non-standard Yes Apple Xcode
AWS, BitBar,
BuddyBuild
Closed
EarlGrey iOS Native
Objective-C,
Swift
Non-standard Yes Google Xcode BuddyBuild Semi OSS
Detox iOS, Android* Native JavaScript Non-standard No
EarlGrey,
Espresso
N/A N/A Full OSS
Xamarin.UITest iOS, Android
Native,
Hybrid?
C# Non-standard No
Microsoft /
Calabash
Visual Studio
Xamarin
TestCloud,
AppCenter
Closed
WDA iOS Native
Java, Python, Ruby,
C#, JavaScript,
Objective-C, Swift,
PHP
WebDriver No XCUITest N/A N/A Semi OSS

10. Framework Platforms Modes Languages Protocol
Appium
iOS, Android, macOS,
Windows
Native, Hybrid, Web
Java, Python, Ruby, C#,
JavaScript, Objective-C,
Swift, PHP
WebDriver
Espresso Android Native, Hybrid Java, Kotlin Non-standard
XCUITest iOS Native Objective-C, Swift Non-standard
EarlGrey iOS Native Objective-C, Swift Non-standard
Detox iOS, Android* Native JavaScript Non-standard
Xamarin.UITest iOS, Android Native, Hybrid? C# Non-standard
WDA iOS Native
Java, Python, Ruby, C#,
JavaScript, Objective-C,
Swift, PHP
WebDriver

11. Framework Requires Source?
Underlying Tech /
Vendor
IDE/SDK integration Service Providers Dev
Appium No
WDA, UiAutomator2,
Espresso, WinAppDriver,
appium-for-mac
N/A
Sauce Labs, TestObject,
AWS, BrowserStack, BitBar,
Xamarin, AppCenter, etc…
Full OSS
Espresso Yes Google Android Studio
FireBase, BitRise,
TestObject, BitBar, AWS
Semi OSS
XCUITest Yes Apple Xcode AWS, BitBar, BuddyBuild Closed
EarlGrey Yes Google Xcode BuddyBuild Semi OSS
Detox No EarlGrey, Espresso N/A N/A Full OSS
Xamarin.UITest No Microsoft / Calabash Visual Studio
Xamarin TestCloud,
AppCenter
Closed
WDA No XCUITest N/A N/A Semi OSS

12. Framework Adoption Strongest Value Weak Point
Appium
Most popular open source automation
technology
Write tests in any language, leverage
WebDriver skills, excellent mobile web
support, multi-platform support
Speed, stability*
Espresso Most popular native Android framework
Officially supported by Google. Fastest and
most stable for Android
Single-platform, tests must be written in
JVM language
XCUITest Most popular native iOS framework Officially supported by Apple
Single-platform, tests must be written in
Apple language. Speed, stability.
EarlGrey Low
Officially supported by Google. Benefits of
Espresso for iOS
Project rewrite is in progress. Single
platform.
Detox Low
Writing JS-based tests for React Native
apps
No vendor support. Stability.
Xamarin.UITest Low
Officially supported by Microsoft. Write in
C#.
Speed, stability. Single language.
WDA Low
Write tests in any language, leverage
WebDriver skills
Single-platform. Speed, stability.

13. Appium
Install Appium via NPM or Appium Desktop. Ensure you have system dependencies in
place (Android SDK, adb, Xcode, iOS SDK, etc…)
Write a WebDriver-style test using the Selenium or Appium client in your language.
Start the Appium server, then run your test. It speaks to the Appium server, which
turns WebDriver requests into automation behavior, and returns WebDriver
responses.
The client makes the response available to your test script, which can take action or
perform verification based on the result

14. Espresso
Add Espresso to your Android app’s test dependencies
Write JUnit tests that use Espresso’s view matchers to find elements and interact with
them. The same library contains assertion methods.
Run your test using AndroidJUnitRunner

15. XCUITest
Add a class to your iOS project in Xcode that extends XCTestCase
Override a setUp method and call XCUIApplication’s launch method
Use the object-oriented XCUITest API to find elements and interact with them
Use XCTAssert methods to make verifications
View reports in Xcode’s Test Navigator
🛠

16. EarlGrey
Add a new iOS unit testing bundle to your iOS project
Install the EarlGrey rubygem and add EarlGrey as a CocoaPod dependency
Add a new XCTestCase class which imports EarlGrey
Use methods chained on the EarlGrey library to find elements, perform actions, and
make assertions (just like in Espresso)
View reports in Xcode’s Test Navigator
🛠

17. Detox
Install detox via npm and applesimutils via homebrew
Decorate your React Native project’s package.json with Detox config information
(device to run on, xcodebuild instructions for your app, etc…)
Add a JS test script that utilizes Detox’s global variables which will be injected by the
Detox runner (e.g., device, element)
Build the app using detox build and then run your test using detox test, which wraps
a JS test runner of your choice, e.g., Mocha
🛠

18. Xamarin.UITest
Add a new UITest project to your Visual Studio solution with the appropriate UITest
template. This will make sure the NuGet dependencies are set.
Associate your mobile projects with the test project
Create a C# test class that imports NUnit and Xamarin.UITest
Write a [SetUp] decorated method that calls ConfigureApp.<platform>.StartApp()
(for the appropriate platform)
Use the resulting app object to perform actions on matching predicates
🛠

19. WebDriverAgent
Clone the WebDriverAgent repo and run its bootstrap script to use Carthage and
download dependencies
Open the WebDriverAgent Xcode project and run it, or use xcodebuild to build and
run from the command line
Watch the output of WebDriverAgent’s logs to determine its port
Write a WebDriver-style test using the Selenium or Appium client in your language.
Point the RemoteWebDriver to the WDA host/port, and run the test.
🛠

20. 5 years from now
(in my dreams)

21. Espresso
(Android)
Appium
(Many)
XCUITest
(iOS)
🛠
EarlGrey
(iOS)
🛠
WebDriverAgent
(iOS)
🛠
Xamarin.UITest
(iOS + Android)
🛠
Detox
(iOS + Android)
🛠

22. Highly integrated with other apps,
locations, experiences
Situation-aware, AR
Spanning many form factors
Phone, watch, TV, goggles, glasses,
installations, tables, ?
Spanning many platforms React Native leads the way
More than stock UI components 3d UIs, VR/AR, IoT
The “mobile” app of the future

23. Visual testing
Does my app look the way it’s supposed to?
(AVM*)
Performance testing Is my app experience responsive enough?
Security testing
Is my app safe for my customers to use? Is it safe
for me to let them use it?
Location testing
Does my app work the same way anywhere in the
world?
Flow testing
What are all the paths through my app? How hard
is it for users to get where I want them to go?
Beyond functional testing

24. “Shift left” is now a household term
Automate the automation - code gen, AI, etc…
Tool and framework fatigue
The automation trend continues

25. To InfinityDriver & Beyond…

26. An official extension to the WebDriver protocol
A set of Node.js base classes and libraries for easily writing drivers
Multiple drivers, each one a WebDriver interface on top of the best automation tech of the
day for a given platform, plus a helpful collection of other utilities. A stable API on top of a
changing set of underlying technologies.
A community of practitioners who maintain an ever-expanding knowledge-base for
different platforms
The Appium of the Future / ✩Driver / ∞Driver

27. The WebDriver protocol is a great start! But is it the end of the line?
So many new kinds of testing emerging. Do we create a different protocol for each
one?
Whatever we do, let’s not compete on standards. Implementations? Yes. Standards?
No.
…and beyond!

28. Promote WebDriver over other options. Not because it’s necessarily the best, but
because it’s a standard.
Contribute your wisdom and experience for a particular platform to the community-
maintained Appium driver for that platform.
Contribute code and help maintain! If we all pitch in a little bit, we all get a lot more in
return. This is the standard open source value prop for businesses.
Big visions require broad support
1
2
3

29. Thank You
Jonathan Lipps • Founding Principal • Cloud Grey
@AppiumDevs • @cloudgrey_io • @jlipps

30. Questions?
Jonathan Lipps • Founding Principal • Cloud Grey
@AppiumDevs • @cloudgrey_io • @jlipps
Don’t forget to sign up for
Your free weekly Appium newsletter
https://appiumpro.com