Mobile applications are now a required component of enterprise operations, with both consumers and workers relying on mobile technologies for communications and productivity. To ensure a functional, secure, and worthwhile mobile experience, enterprises must stay abreast of growing complexity in mobile devices, applications, and platforms while remaining responsive to unforgiving user expectations for speed and service. To meet this challenge, many firms are turning to cloud-based automated testing, which reduces the complexity and cost of manual, on-premise testing and offers extraordinary flexibility to accommodate a variety of scenarios. Joe Schulz outlines the reasons why cloud-based application testing is beneficial, discusses the role it plays in supporting testing automation, and explores the best practices for adopting this solution. Get a practical grounding in cloud-based automated mobile testing. Learn how this approach helps companies speed time to market, optimize security and performance, increase user satisfaction, and contain costs.

1. 6/2/15
1
Cloud-­‐Based,
Automated
Mobile
App
Tes5ng
for
the
Enterprise
Joe
Schulz,
AVP
Emerging
Technology
Today’s
Speaker
Joe
Schulz
Orasi
So:ware
AVP,
Emerging
Technology
joe.schulz@orasi.com

2. 6/2/15
2
Today’s
Agenda
z What
is
Mobile
Tes<ng?
z What
is
Cloud
Tes<ng?
z How
do
I
Automate
for
Mobile?
z What
is
Con<nuous
Mobile?
z Ques<ons?
What
is
Mobile
Tes5ng?

3. 6/2/15
3
Mobile
is
BIG
Source:
Forbes
Magazine,
3/23/2014
7
Billion
People
in
the
World
6
Billion
have
a
Mobile
Phone
Only
4.5
Billion
Have
What…?

4. 6/2/15
4
“Only
16%
of
people
will
try
out
an
app
more
than
twice…”
Digital
Trends
Magazine,
March
2013
“80-­‐90%
of
all
apps
are
used
once
and
then
deleted…”
Tech
Crunch
Magazine,
March
2013
Source:
Dimensional
Research,
2014
The
New
Benchmark:
48%
expect
a
mobile
app
to
respond
in
two
seconds
or
less

5. 6/2/15
5
Maintaining
con<nuity
is
a
huge
challenge
Layout
issue
Device
incompaJbility
OS
issue
Network
issue
AuthenJcaJon
issue
Impact
of
the
network
on
test
results
Capacity
>
500+
users
Capacity
<
200
users
By
not
considering
actual
network
condi<ons,
testers
can
get
trapped
Without
real
network
condi<ons
Real
network
condi<ons
VS
•
Performance
tests
under
ideal
lab
condiJons
can
result
in
a
false
sense
of
security
that
an
applicaJon
will
perform
well
aQer
deployment
•
By
accurately
emulaJng
real-­‐world
network
condiJons,
organizaJons
can
improve
the
load
Jme
of
web
assets
to
mobile
devices
by
up
to
80%
while
driving
remediaJon
costs
down
by
30%

6. 6/2/15
6
Why did this happen?
Is it AT&T’s fault?
Is it Apple’s fault?
Is it YOUR fault?
The
Cost
of
Not
Tes<ng
Source:
New
York
Times
Online,
February
2012
“…customers
using
iPads…
actually
paid
twice”
“…problem
began
in
July
but
went
undetected…unRl
December”
“Other
mobile
apps…immune
from
the
problem”

7. 6/2/15
7
What
is
Cloud
Tes5ng?
Introducing
the
Orasi
Mobile
TesRng
Maturity
Model
Is
this
YOUR
mobile
lab?

8. 6/2/15
8
Orasi
Mobile
Tes<ng
Maturity
1:
Handheld
Manual
tesJng
on
borrowed
or
purchased
mobile
devices
5
Stages
of
Mobile
Tes<ng
Maturity
1:
Handheld
˜ Easy
˜ Expensive
˜ No
AutomaJon
2:
Crowdsourcing
Manual
tesJng
by
consumers
in
return
for
a
small
reward

9. 6/2/15
9
5
Stages
of
Mobile
Tes<ng
Maturity
1:
Handheld
˜ Easy
˜ Expensive
˜ No
AutomaJon
2:
Crowdsourcing
˜ Global
˜ Unreliable
˜ No
AutomaJon
3:
Emulators
SoQware
version
of
mobile
OS
running
on
a
workstaJon
5
Stages
of
Mobile
Tes<ng
Maturity
1:
Handheld
˜ Easy
˜ Expensive
˜ No
AutomaJon
2:
Crowdsourcing
˜ Global
˜ Unreliable
˜ No
AutomaJon
3:
Emulators
˜ Cheap
˜ AutomaJon
˜ Unreliable
˜ Market
Delays
4:
Tethered
ConnecJng
physical
devices
to
a
workstaJon
with
USB
or
WiFi

10. 6/2/15
10
5
Stages
of
Mobile
Tes<ng
Maturity
1:
Handheld
˜ Easy
˜ Expensive
˜ No
AutomaJon
2:
Crowdsourcing
˜ Global
˜ Unreliable
˜ No
AutomaJon
3:
Emulators
˜ Cheap
˜ AutomaJon
˜ Unreliable
˜ Market
Delays
4:
Tethered
˜ Real
Devices
˜ AutomaJon
˜ Local
˜ Expensive
5:
Cloud
ConnecJng
to
physical
devices
hosted
by
a
cloud
service
on
a
subscripJon
basis
5
Stages
of
Mobile
Tes<ng
Maturity
1:
Handheld
˜ Easy
˜ Expensive
˜ No
AutomaJon
2:
Crowdsourcing
˜ Global
˜ Unreliable
˜ No
AutomaJon
3:
Emulators
˜ Cheap
˜ AutomaJon
˜ Unreliable
˜ Market
Delays
4:
Tethered
˜ Real
Devices
˜ AutomaJon
˜ Local
˜ Expensive
5:
Cloud
˜ Real
Devices
&
Networks
˜ Emulators
˜ Scalable
˜ Global
Secure
˜ AgnosJc
AutomaJon
˜ Cost-­‐effecJve
˜ Up-­‐Front
Cost

11. 6/2/15
11
Device Cradle
Device Cradle
Perfecto
Mobile
ScriptOnce
AutomaRon
for
Device-­‐AgnosRc
ScripRng
UFT
Mobile
plug-­‐in
for
naRve
HP
ALM
integraRon
Secure
Web
Interface
Internet
or
WAN
Device Cradle
MobileCloud
Service
Mobile
Labs

12. 6/2/15
12
How
do
I
Automate
for
Mobile?
Thousands
of
combina<ons
32
versions
of
Android
released
since
2007
450
acJve
models
of
Android
mobile
phones
21
versions
of
iOS
released
since
2007
7
Common
operaJng
plaborms
5
Major
carrier
Networks
in
the
US
14
primary
cell
phone
manufacturers
worldwide
4
wildly
popular
browser
technologies
12
common
mobile
scripJng
and
programming
and
languages

13. 6/2/15
13
How
Big
Is
The
Challenge?
Manual
Test
Cases
to
Execute
90
iOS
Devices
to
Test
(iPhone
5/5S/5C/6/6S
iPad
2/3/4/Mini)
9
iOS
Versions
to
Test
(iOS
6/7/8)
3
Test
Case
ExecuJons
2,430
Average
Minutes
per
Test
Case
ExecuJon
8
40.5
days
4
162
days
=
8
FTE
146
44
FTE
52
FTE
Total
iOS
Manual
Test
ExecuJon
Time
Per
Build
Testable
Releases
per
sprint
Total
iOS
Manual
Test
Execu<on
Time
Per
Sprint
Android
Device
VariaJons
to
Test
Total
Android
Manual
Test
Execu<on
Time
Per
Sprint
Total
Manual
Execu<on
Time
for
iOS
and
Android
=
x
x
x
Automa<on
Objects
Objects
are
the
building
blocks
of
automaJon
Visual
(OCR)
vs.
NaJve
object
recogniJon
AutomaJon
invokes
methods
and
agributes

14. 6/2/15
14
Hybrid
Mobile
Automa<on
§ Objects
in
Automa<on
w NaJve
objects
preferred
but
limiJng
w Visual
(OCR)
objects
more
flexible
but
problemaJc
§ Best
alterna<ve
is
hybrid
approach
mixing
na<ve
and
visual
objects
to
cover
all
scenarios
w Provides
direct
access
to
applicaJon
objects
w Allows
interacJon
with
the
device
outside
of
the
custom
app
UFT
Mobile
Automa<on
Full
support
for
advanced
UFT
funcJonality
like
run-­‐Jme
parameters
and
data
tables
Device-­‐agnosJc
scripJng
allows
test
reuse
across
devices
Integrated
UFT
reporJng
provides
run
results
with
recordings
and
checkpoint
screen
shots

15. 6/2/15
15
What
is
Con5nuous
Mobile?
Con<nuous
Integra<on
Frequently
merging
all
developer
changes
into
a
shared
mainline
to
avoid
merge
conflicts
Con<nuous
Tes<ng
Applying
test
automaRon
to
the
conRnuous
integraRon
process
Con<nuous
Delivery
Extending
the
automaRon
paradigm
across
the
enRre
so:ware
delivery
chain

16. 6/2/15
16
CI/CT/CD
for
Mobile
Code
IDE
Unit
Test
Build
Tool
Developer
Machine
Test
Repo
Code
Repo
Defect
Repo
Development
Server
Unit
Test
Build
Tool
Deploy
App
CI
Server
Build
Server
Mobile
Server
Test
Tool
Test
Server
CI/CT/CD
for
Mobile
Code
IDE
Unit
Test
Build
Tool
Developer
Machine
Test
Repo
Code
Repo
Defect
Repo
Development
Server
Unit
Test
Build
Tool
Deploy
App
CI
Server
Build
Server
Mobile
Server
Test
Tool
Test
Server
Eclipse
w/ADT
JUnit
Ant
HP
ALM
SVN
Jira
JUnit
Ant
Cloud
Server
Jenkins
Cloud
Server
HP
UFT

17. 6/2/15
17
C2
Automation never lags by more than one sprint
The “s+1” rule of test automation
Sprint
41 Sprint
3Sprint
2Sprint
1 Sprint
5
C1
C Component (non-GUI)
G GUI element
B Business process
Stable
C1
G1
C1
G1
G2
C1
G1
C2
G2
C3
B1
C1
G1
C2
G2
C3
B1
G1
C2
Other
Considera5ons

18. 6/2/15
18
Public,
Private
or
Mixed?
Public
Cloud
Private
Cloud
§ Devices
available
first-­‐come,
first-­‐served
§ Select
specific
handset
as
needed
§ ReservaRons
can
be
made
in
advance
§ Ac<vity
cleansing
important
since
devices
are
shared
§ IniRal
state
not
guaranteed
§ Purchase
blocks
of
minutes
§ Device
“cradles”
reserved
for
secure,
exclusive
use
§ Specify
any
device
to
be
placed
in
cradle
§ Can
“swap”
devices
on
request
§ Automa<on
requires
private
cloud
§ Ac<vity
cleansing
less
important
§ IniRal
state
consistent
with
previous
session
§ Purchase
cradles
by
the
month
How
Many
Do
I
Need?
§ “Must”
Devices
§ 2-­‐5
devices
used
for
essenJal
and
regression
tesJng
§ AutomaJon
is
criJcal
§ “Major”
Devices
§ 6-­‐12
devices
used
for
compaJbility
tesJng
– Vary
models,
specs,
manufacturers,
and
networks
§ AutomaJon
less
useful
but
cost-­‐effecJve
for
key
tests
§ “Market”
Devices
§ 20-­‐40
devices
used
for
acceptance
tesJng
§ Primarily
manual
tesJng

19. 6/2/15
19
Ques<ons?
If
you
have
any
further
quesJons,
please
contact:
Orasi
Mobile
mobiletesJng@orasi.com
Orasi
Somware
At
A
Glance
Our
Company
· Focus
on
enterprise
soQware
quality
management
· 1800+
Customers
· Partners
with
Mercury/HP
since
1994
Our
Technology
· HP
Test
Management
and
AutomaJon
· HP
Performance
TesJng
· HP
Mobile
Center,
NV
· Perfecto
Mobile,
Mobile
Labs
2012
HP
Support
Partner
of
Year
2013
HP
Somware
Partner
of
Year
Our
Services
· ImplementaJon
Services
· Install
and
Mentoring
· Mobile,
SAP,
Security
and
Performance
TesJng
· Onsite,
Rural
or
Offshore
· SoQware
Support