The document discusses open source testing tools for mobile applications. It begins by explaining that mobile testing requires automating both the application and the full user environment. It then reviews five popular open source test frameworks - Selenium, Appium, Calabash, Espresso, and XCTest UI - comparing their suitability for different uses like web, native, and hybrid apps. Key criteria that organizations should consider when choosing a framework include supporting multiple frameworks, flexibility, autonomy, full end-to-end coverage, and unattended reliable testing. The document concludes with a demonstration of a test automation tool called Quantum that integrates various open source frameworks.

1. Boston
Best Practices using open
source for testing
Eran Kinsbruner, @ek121268
Amir Rozenberg, @Amir_Rozenberg

2. SHIFTING TO OPEN SOURCE, BENEFITS
MOBILE OSS MARKET LANDSCAPE,
TOOLS, COMPARISON
DEMO!!!
OSS: BEYOND TESTING
Q&A

3. What is automated
Mobile Web
Automation target
Cannot be
automated
60%-90%20% 30% 100%
The gap we want to address
Degree of automation
Increased complexity and no timeMany teams with varying set of skills
2/8/2017
TestersDevTestersCoders

5. A Day in A Life –Real User Environments Is Essential
Can Any OSS Framework Handle Such Requirements?
Automation ought to cover
the full user environment
not just the Application
Under Test!

6. Which Tool is the best fit for you?
SELENIUM APPIUM CALABASH ESPRESSO XCTEST UI
When looking at today’s open-source mobile test automation landscape, there are five
highly-adopted test frameworks.
Each tool has advantages for your mobile and web testing depending on your needs.

7. Making the Decision
A Comparison
Detailed analysis of each test
automation tool:
Sometimes More Than A
Single Framework is The Right
Answer

8. Selenium
• The best choice for web test automation teams
testing for responsive web design or stand-alone
web sites
• Less suitable for developing unit testing, making
this framework less appealing for developers
• On its own, Selenium core test reports are not
highly informative and lack unique insights

9. Appium
• Best suited for QA teams testing the functionality
of native, mobile web and mobile hybrid apps
across iOS and Android
• Less suitable for developers who wish to develop
and perform unit testing
• Appium reports are a bit limited from a debugging
and fast feedback loop perspective, and do not
include videos, network logs and key vitals
information

10. Calaba.sh
• Designed for organizations that work in behavior-driven
development workflows
• Does not cover the entire mobile system level events
• Offers an easy path to both develop and test features in
parallel, all in an easy user-flow based language
• Appealing for both dev and QA practitioners
• Provides solid insights and reports to both dev and QA
teams

11. Espresso & XCTest UI
• Both requires a separate Test App executable
• Espresso is for Android and XCTest is for iOS – not a single script
• Both tools are fully integrated into development IDEs such as Android Studio/Xcode, and
offer very easy-to-develop techniques, including test recorders
• Fully maintained by Google and Apple, which assures they always support the latest OS
features so developers can stay ahead of the market and test accordingly
• Support both unit testing types and functional UI testing
• Both are app-context only, which limits their abilities to test for user condition scenarios
XCTEST UIESPRESSO

12. Things to Consider
• Lab: Support for variety of frameworks is important for enterprises
• Flexibility
• Autonomy
• Complete E2E coverage
• Unattended reliable testing
• Automation: support for multiple OSS frameworks becomes important
• Different “persona’s” need different tools under 1 lab
• Cross platform scripting
• Robust framework to support high test coverage
• Analysis: Digital reporting is necessary for teams to take action and fix issues faster
& earlier
When choosing a test automation tool

15. Area Components Quantum
Management
Reporting
Execution Manager
Test
Development
Test Authoring
(IDE+Language)
Object Repository
Object Locator/Spy
Appium/Selenium
✓OSS
=
✓OSS
✓OSS
http://projectQuantum.io

16. Digital – Responsive Web Application

17. DEMO – BDD OSS Framework

18. Few other OSS-made mobile testing toys
Production Insight/ ”Testing in Production”
Network features: HAR file & NETEM
Some others:
• Security scans:
https://github.com/MobSF/Mobile-
Security-Framework-MobSF
• Deep visual analysis
• Etc.

19. Q&A
@Amir_Rozenberg@ek121268

20. @Amir_Rozenberg@ek121268

21. 22
Open &
Integrated
Xcode
Selenium