This document discusses challenges and lessons related to automating mobile application testing. It notes that mobile testing faces issues like device fragmentation, difficulty simulating real-world conditions, and maintaining a device lab. The key lessons are to prioritize test cases, automate functional tests on simulators first, then automate critical cases on real devices. Non-mobile elements like data and environments should also be automated. Choosing the right tools, using cloud options, and simulating conditions can help address challenges of eating the "elephant" of full mobile automation.

1. ©2015 InfoStretch Corporation. All rights reserved.
Dr. Ashok Karania | January 21, 2015
Mobile
Automation
Lesson from the Trenches

2. ©2015 InfoStretch Corporation. All rights reserved.
Agenda
§ Mobile
Landscape
&
Tes1ng
Challenges
§ Automa1on
Challenges
§ Lessons
From
The
Trenches
§ Case
Study
§ Q
&
A

3. ©2015 InfoStretch Corporation. All rights reserved.
§ Apps
Tested
§ Employees
100+
§ Apps
developed and
deployed
3000+
Global Footprint
A leading provider of next-gen mobile application lifecycle services
ranging from design and development to testing and sustenance.
About us
Locations
Corporate HQ: Silicon Valley
Offices: Conshohocken (PA), Ahmedabad (India), Pune (India), London (UK)
600+
250K
§ Active Mobile
Customers
Engagements
60+
About Infostretch
§ Real devices
in mobile labs

4. ©2015 InfoStretch Corporation. All rights reserved.
Mobile Landscape &
Testing Challenges

5. ©2015 InfoStretch Corporation. All rights reserved.
25%
1.7B
95%Of adults are online with
3+ devices
Forrester 2013
Number of smartphones and
tablets shipped worldwide will
more than double by 2017
IDC 2014
Of enterprises allow BYOD
in some form
FierceMobileIT2014
31.7%The mobile applications market
is expected to increase
31.7% by 2017
IDC 2014

6. ©2015 InfoStretch Corporation. All rights reserved.
THE FUTURE IS MOBILE
WEARABLES GLASS
FLEXIBLE
SMARTPHONESVIRTUAL

7. ©2015 InfoStretch Corporation. All rights reserved.
AND BEYOND
INTERNET
OF
THINGS
BIONICS
AND
ROBOTICS
COGNITIVE
COMPUTING
3D PRINTERS
TRANSPARENT
LCDs
CONNECTED
CARS

8. ©2015 InfoStretch Corporation. All rights reserved.
YOU NEED A NEW VISION, NOT A NEW VERSION
CONNECTED AGILE COMMUNITY
NEXT GEN ENTERPRISE

9. ©2015 InfoStretch Corporation. All rights reserved.
Mobile Automation Is Imperative
§ Mobile Testing Is Everything We Do On Web & More & More
§ Enables To Meet Compressed Release and Development Cycles
§ Address Fragmentation of Devices, Platforms and Carriers
§ Generate Real World Testing Conditions
§ Improve Overall Testing Quality
§ Overall Time & Resource Efficiencies
§ Systematic

10. ©2015 InfoStretch Corporation. All rights reserved.
Automation Challenges
from the Trenches

11. ©2015 InfoStretch Corporation. All rights reserved.
1. Device Fragmentation
~12K Android
Devices

12. ©2015 InfoStretch Corporation. All rights reserved.
2. Why Real Devices Are A Nightmare
Fragile Slow Maintenance Issues Cost
Other App
Inter-Operability
!

13. ©2015 InfoStretch Corporation. All rights reserved.
3. Simulating Real Life Conditions
The Real Mobile End User Experience Can Be Different
§ Servers Will Be Loaded
§ Devices Will Behave Differently
§ Network Will React Differently
Apps Behave Differently In Special Conditions
§ CPU Usage
§ RAM
§ Network Conditions
§ Other Apps
§ Environment conditions
§ Light Conditions
We Need
Real Life
Testing
Conditions!

14. ©2015 InfoStretch Corporation. All rights reserved.
4. Mobile Device Lab Is A Big Challenge
§ Test Lab Creation
§ Test Lab Maintenance
§ Team
§ Buy Or Cloud
§ Globally Available
§ Management
§ Contracts
§ OS Upgrades
§ Procurement Policy
§ Performance
§ Security

15. ©2015 InfoStretch Corporation. All rights reserved.
Lessons from
the Trenches

16. ©2015 InfoStretch Corporation. All rights reserved.
The best way to deal with mobile device
automa4on challenge is to not
automate
every thing on the mobile devices
”
“

17. ©2015 InfoStretch Corporation. All rights reserved.
How
To
Eat
An
Elephant
§ Priori1ze
Business
Needs
&
Analyze
Test
Cases
§ Design
An
Automa1on
Framework
§ Select
Right
Automa1on
Tool
License
VS
OS,
Integra1ons,
Exis1ng
Investment,
Mobile
Friendly,
Team
Experience
§ Automate
All
Func1onal
Test
Cases
On
Simulators
Mobile
Web
Can
Be
Simulated
By
Tes1ng
On
Real
Browsers
Using
User
Agent
Strings
Na1ve
Can
Be
Tested
By
Using
Simulators
§ Automate
Device
Centric
&
Cri1cal
Func1onal
Test
Cases
On
Real
Devices
§ Iden1fy
&
Integrate
Non-­‐Mobile
Elements
Of
Automa1on
Data
Condi1oning,
SeRngs
Pre-­‐Requisites,
Environment
SeRngs

18. ©2015 InfoStretch Corporation. All rights reserved.
NATURE OF
TEST CASES
Func1onal
Non-­‐
Func1onal
Security
Performance
TYPE OF APP
Business
App
Social
Networking
Gaming
App
U1lity
App
VERIFICATION
CRITERIA
Business
Rules
Look
&
Feel
Response
Time
Content
TARGET
USERS
Enterprises
Consumer
Field
Engineers
Test
Case
Analysis
1.

19. ©2015 InfoStretch Corporation. All rights reserved.
Meeting Device Fragmentation - Device
Type of the App
Games
Social
Business
User Personas
Business
Teen
Social
Geography
Asia Pac
Americas etc.
App Functions
Streaming
Screen Res
GPS etc.
Device Popularity
Form Factor OS
Output: Device/OS/
Test Matrix

20. ©2015 InfoStretch Corporation. All rights reserved.
iOS
Android
HTC Aria
Android
HTC Droid
BB Torch LG CU620
LG
VX9000
Device Implementations for Sign On (10+)
BB 8520
Devices
Browser
Open
Browser
Go To
Sign On
Make
Payment
Confirm
Payment
Sign OffText Verify
Make Payment Test caseQTP/Selenium
Address
ATM
Search
Browser
Open
Select Acct
View
Schedule
Payment
Sync Text
Quick
Links
Nick Name
Browser
Go To
Sign Off
ZIP ATM
Search
Make
Payment
Sync
Image
Confirm
Payment
Framework
Sign On
Component Library
2. Mobile Test Automation Framework

21. ©2015 InfoStretch Corporation. All rights reserved.
Automation Tools
Tool
Supported
Pla.orms
Basis
of
Automa5on
Scrip5ng
Language
Instrumenta5o
n
Required
Repor5ng
Format
Cost
A
Object
Access
All
Major
No
HTML,
Junit,
CSV
Free
P
Both
Na5ve,
Visual
Basic,
Java
Either
HTML,
PDF,
Na5ve
Paid
K
Object
Access
Objec1ve-­‐C
Yes
XCode
Free
C
Object
Access
Ruby,
C#,
JVM
Yes
HTML
Free
MR
Object
Access
Python
Yes
Junit
Free
R
Object
Access
Java
Yes
Junit
Free
MT
Both
JS,
Na1ve
Yes
HTML/Junit
Paid
X
Both
C#,
Ruby
No
HTML
Paid
T
GUI
N.A
Yes
Various
Paid
E
GUI
Na1ve
No
Na1ve
Paid

22. ©2015 InfoStretch Corporation. All rights reserved.
Cloud Options
• Real Devices – Perfecto
• Simulators – Sauce Labs
In-House Mobile Labs
Crowd Sourcing
Meeting The Device Lab Challenge

23. ©2015 InfoStretch Corporation. All rights reserved.
APPROACH QUALITY
COST
EFFECTIVENESS
SECURITY PRIVACY ACCOUNTABILITY
TIME TO
MARKET
DEVICE
COVERAGE
GLOBAL
TESTING
Simulator/Emulator 3 5 3 4 4 4 3 5
Real Mobile Devices 4 2 5 5 5 2 5 4
Remote Mobile Access 4 4 4 4 4 5 4 4
Crowd sourcing 2 3 2 1 1 3 4 4
Managed Testing Services 5 3 5 5 5 4 5 4
Guide:
Least Effective 0 Most Effective 5
Comparative Evaluation Of Device Testing Options

24. ©2015 InfoStretch Corporation. All rights reserved.
Create Real Life Conditions
Manage CPU Usage Manage Memory Manage Network BandwidthCreate & Manage Interruptions
Manage Location Of DeviceManage OrientationTurn WIFI On or Off

25. ©2015 InfoStretch Corporation. All rights reserved.
Summary: How To Eat An Elephant
One Bit At A Time
§ Prioritize Business Needs & Analyze Test Cases
§ Design An Automation Framework
§ Select Right Automation Tool
§ Automate All Functional Test Cases On Simulators
§ Automate Device Centric & Critical Functional Test Cases On Real Devices
§ Identify & Integrate Non-Mobile Elements Of Automation

26. ©2015 InfoStretch Corporation. All rights reserved.
Continue The Dialogue…
Email:
ashok.karania@infostretch.com
Linkedin:
ashokkarania
Blog:
blog.infostretch.com
Blog:
www.ashokkarania.com
Visit
ATOS
Booth
Today
&
Tomorrow!

27. ©2015 InfoStretch Corporation. All rights reserved.
CONNECT WITH US:
Thank You
Tel: 408.727.1100
Email: info@infostretch.com
URL: www.infostretch.com