How do you create a test infrastructure that allows automated testing of both native and hybrid apps? Ankit Desai and Binod Pant will describe the approach they took to develop a comprehensive test infrastructure. Their aim was to leverage MathWorks’ existing home-grown continuous integration system and to provide consistent test tooling across both the iOS and Android platforms. Hear how Ankit and Binod covered mobile use cases and testing requirements; created tools to request and check out the multiple mobile configurations from the hardware/device farm; used a toolset to control the emulators and simulators; and implemented a consistent Java API for authoring tests. They discuss the challenges their teams faced providing emulators with custom configurations and explore various tiers of tooling developed around Appium, ADB, and other tools. See how you can create a framework that looks and feels consistent across iOS and Android, and implement your own native and hybrid automated testing infrastructure.

1. T2
Track Session
4/21/2016 11:00 AM
"Test Infrastructure for Native and
Hybrid iOS and Android
Applications"
Presented by:
Ankit Desai & Binod Pant
MathWorks, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 · 904-278-0524 · info@techwell.com · www.techwell.com

2. Ankit Desai
MathWorks, Inc.
At MathWorks Ankit Desai leads the MATLAB hardware team, mobile, Internet
of Things, and test and measurement quality engineering team that focuses on
architecting and authoring automated tests and test tools. In his spare time, Ankit
teaches IoT and robotics courses at local maker space. His hobbies include
learning more about the web, mobile and tech development, videography, and
photography.
Binod Pant
MathWorks, Inc.
A test automation infrastructure engineer at MathWorks, Binod Pant’s team
specializes in building automation tools for web and mobile. Binod comes with
previous experience in developing web applications and database-backed
applications. In his spare time, Binod likes to learn about new tools and tech
developments, follow developments in physics, play music, and read books.
Binod and his family reside in Boston.

3. 3/16/2016
1
1© 2016 The MathWorks, Inc.
Test Infrastructure for Native and Hybrid
iOS and Android Applications
Ankit Desai and Binod Pant
MathWorks, Inc.
2
MathWorks at a Glance
Earth’s topography
on a Miller cylindrical
projection, created
with MATLAB and
Mapping Toolbox.
● Headquarters:
Natick, Massachusetts U.S.
● Other U.S. Locations:
California; Michigan;
Texas; Washington, D.C.
● Europe:
France, Germany, Italy,
Netherlands, Spain, Sweden,
Switzerland, United Kingdom
● Asia-Pacific:
Australia, China, India,
Japan, Korea
● Worldwide training
and consulting
● Distributors serving more
than 20 countries

4. 3/16/2016
2
3
Core MathWorks Products
The leading environment for
technical computing
 The industry-standard, high-level
programming language
for algorithm development
 Numeric computation
 Parallel computing, with multicore and
multiprocessor support
 Data analysis and visualization
 Toolboxes for signal and image
processing, statistics, optimization,
symbolic math, and other areas
 Tools for application development
and deployment
 Foundation of MathWorks products
4
Core MathWorks Products
The leading environment for modeling,
simulating, and implementing dynamic
and embedded systems
 Linear, nonlinear, discrete-time,
continuous-time, hybrid, and
multirate systems
 Foundation for Model-Based Design,
including multidomain system modeling,
real-time testing, automatic code generation,
and verification and validation
 Open architecture for integrating models
from other tools
 Applications in controls, signal processing,
communications, physical modeling, and
other system engineering areas

5. 3/16/2016
3
5
Additional MathWorks Products
6
MATLAB Mobile

6. 3/16/2016
4
7
iPhone, iPad, & Android Support from Simulink
8
Teams Working on Mobile Platform

7. 3/16/2016
5
9
Lack of support from Continuous Integration system
Lack of uniform testing API across mobile platforms
Lack of access to different configurations
Frequent test updates due to underlying tool change
Testing Pain Points
10
Test Framework

8. 3/16/2016
6
11
Utilize in-house Continuous
Integration system
Provide consistent and reusable
API
Shield test authors from
underlying tool upgrade/change
Automate provisioning of multiple
device configurations
Test
Framework
Goals for Test Framework
12
MathWorks Continuous Integration System
SubmitCode
Build
Test
Identify Test Requirements
Lease machine with specified
requirements
Run tests
Release leased machine
Report

9. 3/16/2016
7
13
Test Requirements
Test
Requirements
Test Platform
Software
Required
Hardware
Required
14
#1 - Preconfigured VMs/containers with mobile
configurations
#2 - External services
#3 - Mobile configuration set up during testing
Possible Solutions

10. 3/16/2016
8
15
Preconfigured VMs/containers with mobile configurations
Idea:- Docker containers with preinstalled
emulators/simulators
Example:- Internally deployed Docker instances
Challenge:- CI system does not support
containers
Verdict:- Consider for future solution
Possible Solution #1
16
MathWorks Continuous Integration System
SubmitCode
Build
Test
Identify Test Requirements
Lease machine and mobile configurations with
specified requirements
Run tests
Release leased machine and mobile
configuration
Report

11. 3/16/2016
9
17
Use external services
Idea:- Invoke external tool from CI system
Example:- Sauce Labs, BrowserStack etc.
Challenge:- Incorporating external tool into
existing CI system is non-trivial
Verdict:- A possible future direction, but more
likely as a complementary approach
Possible Solution #2
18
MathWorks Continuous Integration System
SubmitCode
Build
Test
Identify Test Requirements
Lease machine with correct
requirements
Start external service/tool
Run tests
Stop external service/tool
Release leased machine
Report

12. 3/16/2016
10
19
Set up a mobile fixture during testing
Idea:- Provision device/emulator/simulator during
test (just-in-time)
Example:- Existing setup that allows low cost
hardware testing (Arduino, Raspberry Pi)
Challenges:- Applicable to local resources only,
but extensible to remote resources
Verdict:- Plugs well into the existing CI system
Possible Solution #3
20
MathWorks Continuous Integration System
SubmitCode
Build
Test
Identify Test Requirements
Lease machine with correct
requirements
Lease correct mobile
configuration
Run tests
Release leased machine
Release mobile configuration
Report

13. 3/16/2016
11
21
Just-In-Time Mobile Configuration
Test Framework
22
Device/emulators as a ‘fixture’ during test
Test
Requirements
Device Lease
DeviceManager
Setup/Initialize
MobileSession
Java/MATLAB
API
Test Methods
(testpoints)
Device
Release

14. 3/16/2016
12
23
Device/emulators as a ‘fixture’ during test
Test
Requirements
Device Lease
DeviceManager
Setup/Initialize
MobileSession
Java/MATLAB
API
Test Methods
(testpoints)
Device
Release
24
Test Requirements
Test
Requirements
Test Platform
Software
Required
Hardware
Required

15. 3/16/2016
13
25
Test Requirements
Test
Requirements
Maci64
XCode 7
iPhone 6
iOS-9.2
26
Device/emulators as a ‘fixture’ during test
Test
Requirements
Device Lease
DeviceManager
Setup/Initialize
MobileSession
Java/MATLAB
API
Test Methods
(testpoints)
Device
Release

16. 3/16/2016
14
27
Device Lease/Release
Current
approach
• Connected directly to the test hosts
• Leased along with the test host
• Works out of the box with CI system
Future
approach
• Maintain a separate pool of devices
• Decouple device leasing process
• Select device via Test Requirements
28
Device/emulators as a ‘fixture’ during test
Test
Requirements
Device Lease
DeviceManager
Setup/Initialize
MobileSession
Java/MATLAB
API
Test Methods
(testpoints)
Device
Release

17. 3/16/2016
15
29
DeviceManager
DeviceManager
(Java)
Appium
• Android Emulator API level 22
• Android Emulator API level 23
• iPhone 6 – 9.2
• iPhone 6 – 9.1
• iPad Air – 9.2
• iPad Air – 9.1
• …
iPhone 6 iOS-9.2
30
MobileSession Interface
Test
Requirements
Device Lease
DeviceManager
Setup/Initialize
MobileSession
Java/MATLAB
API
Test Methods
(testpoints)
Device
Release

18. 3/16/2016
16
31
MobileSession
• Input = app, Output = automation object
Entry point to automation API
• Consistent API for Android/iOS
Java and MATLAB APIs
• Tests don’t need to deal with device handling
Removes complexity from tests
32
Device/emulators as a ‘fixture’ during test
Test
Requirements
Device Lease
DeviceManager
Setup/Initialize
MobileSession
Java/MATLAB
API
Test Methods
(testpoints)
Device
Release

19. 3/16/2016
17
33
Test Methods (testpoints) – Java API
@Rule
public MobileSession session = MobileSession.forApp(getAppPath());
@Test
public void clicksAnElement() throws Exception {
session.findElement(By.id("An Element")).click();
}
34
Test Methods (testpoints) – MATLAB API
methods (TestMethodSetup)
function createMobileSession(test)
test.Session = test.applyFixture(MobileSession.forApp(test.AppPath));
end
end
methods (Test)
function clicksAnElement(test)
test.Session.findElement(By.id('An Element')).click();
end
end

20. 3/16/2016
18
35
Test Result
Test Name Result Platform Configuration
mobileLogin.m Passed Win64 Android Emulator API Level 22
mobileLogin.m Passed Maci64 iPhone 6 iOS-9.2
mobileRegister.m Failed Win64 Android Emulator API Level 22
mobileRegister.m Passed Maci64 iPhone 6 iOS-9.2
36
Utilize in-house Continuous
Integration system
Provide consistent and reusable
API
Shield test authors from
underlying tool upgrade/change
Automate provisioning of multiple
device configurations
Test
Framework
Goals for Test Framework

21. 3/16/2016
19
37
?