Regarding the automation of mobile applications, we have two kinds of choices. The first
is going with native automation frameworks such as Espresso for Android and XTCUITest for iOS applications, or going with non-native tools such as Selenium-based tooling (Appium, Selendroid, Robot Framework, many others), Calabash or also many other paid frameworks.
I want to clarify the differences between native and non-native toolings for mobile UI automation. Both tooling ecosystems have advantages and disadvantages, so people are confused about which is best for them.
Non-native tools such as Appium that dominated mobile automation for a long time and people have gained a lot of experiences with it so they want to continue using Appium, also Calabash which is older than Appium and it was also very popular for some time. On the other hand, these tools always need to deal with the latest update coming from the latest versions of the Android and the iOS ecosystem. If not those tools would be in history. For example, Calabash almost became useless for the latest versions after Microsoft acquired it and they stopped supporting it.
4. Mesut Güneş
www.testrisk.com
■ Test automation
■ Mobile testing
■ Mobile automation tools
■ Native mobile automation tools
■ Benefits of native tooling
TALK INCLUDE THESE TOPICS
5. Mesut Güneş
www.testrisk.com
TABLE OF
CONTENTS
MOBILE
AUTOMATION
ECOSYSTEM
What we have for automation
NON-NATIVE
SOLUTIONS
How non-native tools work
01
03
02
04
05
06
NATIVE
SOLUTIONS
How native tools work
BENEFITS: HANDLING
OBJECTS
What things are handled differently
BENEFITS:
PERFORMANCE
How we can maximize the
benefits
BENEFITS: TEAM
BUILDING
What kind of benefits we
can have
7. Mesut Güneş
www.testrisk.com
Mobile applications are packages/binaries that can be installed to the users mobile
devices so that you can let them do what you want.
Mobile applications testing is installing the packages to the target devices and
performing test steps through the application in the devices.
Automation means that all the thing required for manual testing are done by
tools and scripts.
So mobile applications automation requires the followings:
● Application packages (apk / app / ipa)
● Mobile devices/emulator/simulator (Android / iOS)
● Networks
● Permissions
● etc.
WHAT IS MOBILE APPS AUTOMATION
8. Mesut Güneş
www.testrisk.com
What tests in what level we should test the apps.
TEST TYPES AND LEVELS
Test Level
Test Types
Functional Nonfunctional
Unit unit tests
load, stress, security, code
coverage (metric)
Integration
unit integration tests,
component
integration tests,
microservices
testing
load, stress, security, contract
test
System end-to-end test
load, stress, security, reliability,
maintainability, scalability
Acceptance
uat test, alpha - beta
testing
load, stress, security, usability,
AB test
9. Mesut Güneş
www.testrisk.com
Identify the purpose of the automation to get a better result
Automate everything that are possible and meaningful
● Classes/Methods
● Integrations
○ Unit
○ Component
○ System
● Requirements
○ Functional
○ Non-functional
● Processes
○ Development
○ Testing
○ Distributing
WHAT TO AUTOMATE
10. Mesut Güneş
www.testrisk.com
Regarding the the end-to-end testing automation, it can be done by the tools that can
interact to the element of the application inside the devices.
Basically we can classify these tooling as follows
● Native toolings
○ Available inside the development framework
● Non-native tooling
○ External tools that make automation possible
HOW TO AUTOMATE
12. Mesut Güneş
www.testrisk.com
Non-native mobile automation tools are external tools and interact with the mobile
application object by a server connected to the mobile apps that are running inside a
mobile device/simulator/emulator.
Non-native tooling requires a server to translate automation commands to actions on
apps for black-box testing
● Depends on server-client communication
● Depends on some native libraries that are always upgraded by manufacturer
● Totally black-box testing technique
WHAT IS NON-NATIVE TOOLING
13. Mesut Güneş
www.testrisk.com
Non-native toolings work with http server to interact with mobile objects
Interaction can be done by two ways
● Interact with instrumentation/UI library
○ Espresso / UIAutomator / UIAutomator2
○ Robotium
○ XCUITests / UIAutomation
● Image recognition
○ Server take screenshot and find the relevant part of the screen
HOW NON-NATIVE TOOLS WORK
14. Mesut Güneş
www.testrisk.com
APPIUM
Appium is one of the most popular mobile automation solution.
Its popularity comes from the following advantages:
● Familiarity with the Selenium Webdriver
● Supports most of the popular languages
● Supports multiple platforms such as Android, iOS, Windows
● Easily extent one automation code base for all projects
15. Mesut Güneş
www.testrisk.com
APPIUM ARCHITECTURE
Appium is a non-native automation tool whose drivers use native automation
libraries in the http server.
Image: https://digital.ai/catalyst-blog/what-is-appium-server-a-complete-end-to-end-guide
16. Mesut Güneş
www.testrisk.com
CALABASH
Calabash is another mobile automation tool for black-box testing. It was very
popular until 2016 but now it almost become history.
Its popularity comes from the following advantages:
● Behaviour Driven test framework for Mobile automation
● Open-Sourced project: Calabash-android and Calabash-ios
● Supports iOS and Android and Hybrid Apps (Webviews) - (Cross-platform)
● Supports simulators / emulators / real devices
● Supports CI / CD and Clouds - (CLI tools)
● Ruby Based
● Integrated Cucumber
● Calabash Console for real time automation development
17. Mesut Güneş
www.testrisk.com
CALABASH ARCHITECTURE
Calabash is also a non-native automation tool. Android version uses Robotium
which is an external automation library and iOS version uses XCUITest that is a
native framework.
Calabash-Android Calabash-iOS
19. Mesut Güneş
www.testrisk.com
Native mobile automation toolings are integrated libraries to the mobile application
development framework so that developers can write UI tests for the products
under development.
Native toolings are part of the development frameworks so it is guarantee that the
tools are always up to date with the development frameworks and upcoming new
features.
Since we are talking about two major mobile device operating system, we are just
focus on the following platforms:
● Android
○ Espresso
● iOS
○ XCUITest
WHAT IS NATIVE TOOLING
20. Mesut Güneş
www.testrisk.com
HOW NATIVE TOOLS WORK
Native tools works natively, which means that they are providing direct methods via
APIs to use inside the development project globally.
Interactions are made by directly calling the UI objects inside project architecture
by the APIs.
So these are the main benefits:
● APIs know the state of the objects
● We are free to use tool as black-box as well as
contributing the development code as white-box
● We are free to reach the UI objects
● If the app is the Matrix, native tools are the agents
inside the matrix
21. Mesut Güneş
www.testrisk.com
ESPRESSO - WHAT
Espresso is a test automation tool developed by Google for Android UI Tests.
Espresso is open-source, very stable and there many frameworks have been
developed around Espresso.
● It has full feature-set of Junit that works with Espresso
● Android studio provides everything you need
● It provides white-box testing
23. Mesut Güneş
www.testrisk.com
XCUITEST - WHAT
XCUITest stands for Xcode User Interface Testing and it is UI automation
framework iOS applications. XCUITest is developed on top of XCTest and
Accessibility. It is launch by Apple in 2015. It supports all iOS functionality in
terms of testing them by white-box testing techniques.
● XCUITest is native and inside the app source code with different target
● XCode provides everything you need
● It provides white-box testing
● XCTest provides everything you need for testing capabilities
● Accessibility provides UI interaction for all kind of XCode product
26. Mesut Güneş
www.testrisk.com
Native toolings are inside the projects so no need to worry about the use external tool
to get the related objects
Espresso and XCUITest provides the following benefits
● APIs provide methods for all kind of object
● Methods are always up-to-date with latest improvements/updates
● Freely update the source code for the sake of ui-test
NATIVELY AVAILABLE OBJECTS
29. Mesut Güneş
www.testrisk.com
The performance is very important when you grow and have to handle hounderents
tests. The main point for the performance is that how much time your tests require for
running them locally and in a cloud services
● Simple UI frameworks interact with the objects so no need to translate with
servers
● Native tooling know the state of the elements so no need to check them
regularly
● Automation code is inside the test package so no need to handle it
differently while testing on the cloud. This reduce the time drastically
SERVERS --- NO
30. Mesut Güneş
www.testrisk.com
Simple test to compare performance of the tools shows that native toolings are always
better than Appium. This was done by TestProject engineers.
Reference: https://blog.testproject.io/2020/04/08/fastest-mobile-automation-testing-tool-appium-vs-
xcuitest-vs-testproject-vs-uiautomator-vs-espresso/
COMPARE THE PERFORMANCE
32. Mesut Güneş
www.testrisk.com
Better performance from an automation project can be only get when the developers
and testers work cooperatively. Developers are are always result oriented people so
they focus on the development of the features, but the QAs/SDETs must be keen on
the quality. This quality not only the product quality but also quality of the tests,
process, and the documentation. This can be achieved by cross-functional-team
building.
Cross-functional-team requires the followings:
● Team spirit
● Shared common knowledge, every team members
should know basics about
○ What is being built
○ How the things are developed
○ How the things are being tested
● Absence of one person from a role can be filled by from
another role
○ QA/SDET can be developer
○ Developer can be QA/SDET
BUILD TEAM AROUND QUALITY
33. Mesut Güneş
www.testrisk.com
How I am lucky to get those feedbacks from experienced mobile developers :)
● DEV provides testable features
● QA/SDET writes test
● When QA/SDET get familiarity with whole project, s/he can update/add missing
parts for deterministic test
● Dev are ready for making test framework better :)
BUILD TEAM AROUND QUALITY
34. Mesut Güneş
www.testrisk.com
CREDITS: This presentation template was created by Slidesgo, including
icons by Flaticon, and infographics & images by Freepik.
Please keep this slide for attribution.
Does anyone have any questions?
THANKS
References:
● https://developer.android.com/training/testing/espresso
● https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual
/testing_with_xcode/chapters/09-ui_testing.html
● https://appium.io/docs/en/about-appium/intro/
● https://www.browserstack.com/guide/getting-started-xcuitest-framework
● https://www.browserstack.com/guide/appium-vs-espresso
● https://blog.testproject.io/2020/04/08/fastest-mobile-automation-testing-tool-appium-
vs-xcuitest-vs-testproject-vs-uiautomator-vs-espresso/
● https://devexperts.com/blog/ios-ui-testing-frameworks-performance-comparison/
Editor's Notes
We mainly focus on doing the functional tests on system and acceptance level but we can also check that the performance of the apps during this test by keeping all dependent parts (such as network, backend services, devices, test data, test steps) stable so that we can catch the deviation in the performance of the app.
A simple automation code creates a request with a json-wire protocol to the Appium server then Appium server gets the capabilities and creates a driver accordingly then starts to connect to devices/emulators/simulators to interact with the Bootstrap framework inside the target devices and finally the Bootstrap tries to complete the requests.
Once the requests completed or failed for any reasons it sends back the log to the client so there is mutual communication during execution of the simple automation command.
Appium created based on the Selenium Webdriver that was created for web automation so making this technology for mobile solution requires a lot of engineering. Appium includes lot of development to cover many edge cases and it will be also updated/refactored by upcoming updates in native automation frameworks, Espresso and XCUITest, that are updated every year by Google and Apple. As a result, Appium is an over-engineered tool.
Same as in the Appium, a simple automation command that is initiated by step definition send requests to the server inside the devices so it can interact with the app UI objects. This interaction is provided by Robotium for android and by XCUITest for iOS.
There is also a detail that the iOS server should be inside the build app which means that you need to rebuild iOS app for the Calabash target.
The main differences between Appium and Calabash is that the server is inside the devices for the Calabash and you are sending query for the UI objects.
Talk about followings:
What native means
Where app code, UI code, unit test code
How to apply white-box testing
How Espresso is initiated
How Espresso can be architectured
POM
Talk about followings:
What native means
Where app code, UI code, unit test code
How to apply white-box testing
How Espresso is initiated
How Espresso can be architectured
POM