Your company has decided to move into the 21st century and is developing a shiny new mobile app. But you don’t know where to start. How many devices do you need to test? Can you take existing tests and modify them? How do you account for conditions such as loss of connectivity, a virtual keyboard, and new user interactions? What about the questions you don’t even know to ask? Join Bambi Rands as she introduces you to the world of mobile testing. Learn from Bambi’s first-hand experience testing mobile apps and building a mobile test strategy. Discuss the shift from testing web software to testing mobile apps. Review the use of personas and test scenarios vs. step-by-step scripts. Decide which devices to start using and how to increase your coverage of screen size, OS, and networks as your mobile test organization grows. Increase your testing skills and leave with a better understanding of how to account for the many mobile-specific nuances you will encounter.

1.
T10
Mobile
Testing
10/5/17
11:15
Mobile
Testing:
Where
to
Start
Your
Journey
Presented
by:
Bambi
Rands
Mutual
of
Omaha
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2.
Bambi
Rands
Mutual
of
Omaha
At
Mutual
of
Omaha
Bambi
Rands
is
a
quality
assurance
lead
with
a
focus
on
mobile
testing.
Leveraging
her
prior
experience
on
the
business
side
of
insurance,
she
has
spent
the
past
fifteen
years
creating
testing
solutions
in
roles
of
business
analyst,
consultant,
trainer,
and
quality
assurance
lead.
With
real-­‐world
experience
helping
a
Fortune
500
company
establish
an
enterprise
test
framework
and
SDLC
process,
creating
training
material,
and
providing
coaching
support,
Bambi
was
uniquely
qualified
to
usher
Mutual
of
Omaha
into
the
mobile
era
by
mentoring
personnel
and
creating
mobile-­‐specific
strategies,
processes,
device
management
protocols,
and
training
materials.

3. 1
Mobile
Tes*ng:
Where to
Start Your
Journey
1
Agenda
• Introduc+ons
• Interac+ve
Session
• Shi3
from
Web
to
Mobile
App
Tes+ng
• Review
the
Use
of
Personas
• Test
Scenarios
vs.
Step-­‐By-­‐Step
Scripts
• Mobile
Test
Scenario
• Device
Selec+on
• Mobile
Tes+ng
Approach
• Bug
Documenta+on
• Automa+on
(Test
ID)
• Miscellaneous
• Ques+ons
2

4. 2
Introduc*ons
My
Bio
-­‐
At
Mutual
of
Omaha
Bambi
Rands
is
a
quality
assurance
lead
with
a
focus
on
mobile
tes+ng.
Leveraging
her
prior
experience
on
the
business
side
of
insurance,
she
has
spent
the
past
fi3een
years
crea+ng
tes+ng
solu+ons
in
roles
of
business
analyst,
consultant,
trainer,
and
quality
assurance
lead.
With
real-­‐world
experience
helping
a
Fortune
500
company
establish
an
enterprise
test
framework
and
SDLC
process,
crea+ng
training
material,
and
providing
coaching
support,
Bambi
was
uniquely
qualified
to
usher
Mutual
of
Omaha
into
the
mobile
era
by
mentoring
personnel
and
crea+ng
mobile-­‐
specific
strategies,
processes,
device
management
protocols,
and
training
materials.
Group
–
Mobile
App
tes+ng?
Responsive
Web
tes+ng?
3
Interac*ve Session
Let’s
learn
from
each
others
experiences
Feel
free
to
add
to
the
conversa+on
on
what
you
have
learned
4

5. 3
Shi> from Web to Mobile App Tes*ng
Example
of
what
to
consider...
We
are
crea+ng
a
mobile
app
for
snow
skiing
resort.
We
want
to
let
the
users
know
the
current
weather,
the
condi+ons
of
the
slopes
and
slopes
loca+on
based
on
the
users
posi+on,
and
snow
reports.
What
are
some
of
the
usability
situa+ons
that
we
should
consider?
Kno[,
D.
(2015)
Hands-­‐on
Mobile
App
Tes1ng.
New
York,
NY:
Addison-­‐Wesley.
5
How should we test these scenarios?
We
have
to
test
the
app
the
way
the
user
will
be
using
it.
• Does
the
user's
device
work
in
the
cold?
Test
outside
at
different
temperatures
• Does
the
user's
device
get
a
connec+on
on
the
top
of
the
mountain
vs
in
the
valley?
Test
on
the
top
of
the
mountain
and
in
the
valley
• Is
the
tap
area
of
the
bu[ons/icons
able
to
be
used
with
gloves
on?
Test
while
wearing
the
Touchscreen
gloves,
large
fingers
vs
small
fingers
• Can
you
locate
the
user
if
they
are
in
mo+on?
Test
going
up
the
ski
li3,
view
while
moving
at
top
of
mountain,
in
a
car
in
the
resort
parking
lot
• What
will
happen
when
the
user
has
a
Wi-­‐Fi
connec+on
at
the
bo[om
of
the
ski
slope
but
loses
it
going
up
the
li3
while
using
our
app?
Does
your
app
crash
or
s+ll
work?
Test
going
up
the
ski
li3
OR
turn
off
Wi-­‐Fi
• Is
the
weather
at
the
ski
lodge
different
than
at
the
top
of
the
mountain?
Test
on
a
day
where
the
weather
condi+ons
are
different
• Does
the
app
work
while
the
user
is
on
the
way
to
the
ski
slope
so
they
can
review
the
weather
condi+ons?
Test
in
Airplane
mode
6

6. 4
Use of Personas
Personas
are
a
great
way
to
understand
who
your
user
base
is
and
how
to
design
and
ul+mately
test
your
app.
• 70
year
old
who
has
a
PC
and
Samsung
smart
phone
• 21
year
old
who
has
a
iPad,
iPhone
and
an
Apple
watch
Depending
on
your
target
user(s)
the
personas
will
help
you
determine
test
scenarios
to
create
and
on
what
devices
to
test
the
app
with.
Ski
app….
Who
will
use
the
app
and
which
type
of
devices?
7
Test Scenarios vs. Step-­‐By-­‐Step Scripts
In
the
past
I
have
always
trained
to
include
step-­‐by-­‐step
tests
so
that
any
tester
off
the
street
can
pick
up
a
test
and
run
it.
In
mobile,
the
app
should
be
designed
to
be
intui+ve.
So,
we
have
moved
to
test
scenarios.
•
STORY
–
I
need
to
be
able
to
view
the
current
weather
so
I
can
know
how
to
dress
while
skiing
• Verify
the
weather
map
displays
with
current
radar
• Verify
ability
to
use
pinch
and
zoom
• Verify
the
home
bu[on
takes
me
to
the
home
page
Mobile
is
fast
pace
and
no
need
or
+me
to
write
step-­‐by-­‐step
scripts
8

7. 5
Mobile Test Scenarios
I
started
with
informa+on
from
h[p://www.guru99.com/tes+ng-­‐mobile-­‐apps.html
• Func+onal,
Performance,
Security,
Compa+bility,
Recoverability
and
Usability
Tes+ng
Func+onal
–
examples
• To
validate
whether
the
applica+on
goes
into
minimized
mode
whenever
there
is
an
incoming
phone
call.
In
order
to
validate
the
same
we
need
to
use
a
second
phone,
to
call
the
device.
• To
validate
whether
the
phone
is
able
to
store,
process
and
receive
SMS
whenever
the
app
is
running.
In
order
to
validate
the
same
we
need
to
use
a
second
phone
to
send
SMS
to
the
device
which
is
being
tested
and
where
the
applica+on
under
test
is
currently
running.
• To
validate
that
the
device
is
able
to
perform
required
mul+tasking
requirements
whenever
it
is
necessary
to
do
so
9
Mobile Test Scenarios
Performance–
example
• To
validate
whether
the
response
+me
of
the
applica+on
is
as
per
the
requirements.
• To
evaluate
product
and/or
hardware
to
determine
if
it
can
handle
projected
load
volumes.
• To
evaluate
whether
the
ba[ery
life
can
support
the
applica+on
to
perform
under
projected
load
volumes.
• To
validate
applica+on
performance
when
network
is
changed
to
WIFI
from
3G/4G
or
vice
versa.
10

8. 6
Mobile Test Scenarios
Security
–
examples
• To
validate
that
the
applica+on
has
a
strong
password
protec+on
system
and
it
does
not
permit
an
a[acker
to
obtain,
change
or
recover
another
user’s
password.
• To
validate
that
the
applica+on
does
not
suffer
from
insufficient
session
expira+on.
• To
iden+fy
the
dynamic
dependencies
and
take
measures
to
prevent
any
a[acker
for
accessing
these
vulnerabili+es.
11
Mobile Test Scenarios
Compa+bility
–
examples
• To
validate
that
the
user
Interface
of
the
applica+on
is
as
per
the
screen
size
of
the
device,
no
text/control
is
par+ally
invisible
or
inaccessible.
• To
ensure
that
the
text
is
readable
for
all
users
for
the
applica+on.
• To
ensure
that
the
call/alarm
func+onality
is
enabled
whenever
the
applica+on
is
running.
The
applica+on
is
minimized
or
suspended
on
the
event
of
a
call
and
then
whenever
the
call
stops
the
applica+on
is
resumed.
12

9. 7
Mobile Test Scenarios
Recoverability
–
examples
• Crash
recovery
and
transac+on
interrup+ons
• Valida+on
of
the
effec+ve
applica+on
recovery
situa+on
post
unexpected
interrup+on/crash
scenarios.
• Verifica+on
of
how
the
applica+on
handles
a
transac+on
during
a
power
failure
(i.e.
Ba[ery
dies
or
a
sudden
manual
shutdown
of
the
device)
13
Mobile Test Scenarios
Usability
–
examples
• To
ensure
that
the
valida+on
for
the
tapping
zoom-­‐in
and
zoom-­‐out
facili+es
should
be
enabled.
• To
ensure
that
the
keyboard
input
can
be
minimized
in
an
appropriate
manner.
• To
ensure
that
the
applica+on
provides
a
method
for
going
back
or
undoing
an
ac+on,
on
touching
the
wrong
item,
within
an
acceptable
dura+on.
• To
ensure
that
the
contextual
menus
are
not
overloaded
because
it
has
to
be
used
quickly.
14

10. 8
Mobile Test Scenarios
We
added
to
the
list
as
we
came
across
addi+onal
things
like:
• Verify
that
field
tool
func+ons
should
or
should
not
be
turned
on
(Copy,
Paste,
Select
All)
• Verify
that
Autocorrect
should
or
should
not
be
turned
on
(Search
for
loca+on
–
town
names
were
autocorrec+ng)
• Verify
camera
func+onality
–
Ligh+ng
(Test
in
brightly
lit
area,
standard
room
ligh+ng,
dimly
lit
area,
black
background,
gray
background,
and
white
background.)
15
Device Selec*on
• Start
with
one
iPhone,
one
iPad,
one
Android
phone
and
one
Android
Tablet.
• ONLY
upgrade
to
the
newest
OS
on
one
iOS
device
and
one
Android
• Once
you
start
gekng
more
devices
they
will
be
the
newest
OS
so
leave
your
others
at
older
versions
• Increase
#
of
devices
as
you
progress
through
the
Tes+ng
cycle
(see
Mobile
Tes+ng
Approach)
• Quarterly
Review
-­‐
research
what
is
available
in
the
market
and
the
device
&
opera+ng
system
usage
for
your
current
apps
to
determine
when
to
upgrade
OS
or
phase
out
a
device
166

11. 9
Mobile Tes*ng Approach
Test
Type
Devices
Notes
When
How
Tools
Who
Ux/
Usability
1
device
Process
owned
by
Digital
Services
Throughout
Design
Itera+ons
Camtasia
/
Invision
Digital
team
ADA
Compliance
Review
(Web)
NA
Dev
WAVE
(Responsive
Web)
Developers
Unit
NA
Tes+ng
of
internal
func+onal
components
Dev
Automa+on
Karma
(Unit
Tes+ng)
Junit
(Web
Service
Unit
Tes+ng
and
QA
Framework)
NUnit
(AngularJs)
Fisheye/Crucible
(Code
Reviews)
Ionic,
Jasmine
Developers
177
Test
Type
Devices
Notes
When
How
Tools
Who
Web
Service
API
NA
Tes+ng
of
internal
components
Dev
Manual
/
Automa+on
Junit,
Swagger,
JEB,
Spock
Developers,
Business
QA
Analyst
Security
-­‐Ports
for
entry
to
servers
NA
Open
ports
for
entry
to
the
servers.
Process
owned
by
Security
Include
Security
in
Planning
stage
Security
Integra+on
1
to
2
devices
Mobile
App
to
the
Web
Service
Dev
Emulators
/
Automa+on
/
Manual
/
Browsers
iOS
Simulator,
Android
Simulator,
Ripple,
Selenium/
Appium
Developers,
QA
Analyst
Mobile Tes*ng Approach
188

12. 10
Test
Type
Devices
Notes
When
How
Tools
Who
Manual
tests
to
automate
for
Smoke
Tes+ng
1
to
2
devices
Create
tests
that
check
the
main
parts
of
the
app
are
s+ll
working
then
hand
over
to
the
Automa+on
Engineer
As
the
func+onality
of
each
sec+on
of
the
app
is
in
a
stable
state.
Manual
Test
scenarios
repository
QA
Analyst
if
Automa+on
Engineer
does
not
have
QA
experience
Automated
Smoke
Test
1
to
2
devices
Create
the
suite
of
Smoke
tests.
This
test
suite
may
increase
as
the
new
automated
func+onal
tests
are
completed
for
the
stable
por+ons
of
the
app
Runs
for
only
seconds
or
minutes.
Execute
a3er
every
commit
Automa+on
JUnit
(Web
Service
Unit
Tes+ng
and
QA
Framework)
Selenium
(QA
Automa+on)
Appium
(Device
Automa+on)
Gradle
(Build
Tool)
Automa+on
Engineer
Mobile Tes*ng Approach
199
Test
Type
Devices
Notes
When
How
Tools
Who
Manual
Func+onal
tests
1
to
2
devices
Crea+on
of
test
scenarios.
Determine
the
logical
tests
to
automate
As
the
func+onality
of
each
sec+on
of
the
app
is
in
a
stable
state
Manual
Test
scenarios
repository,
u+lize
physical
devices
QA
Analyst
if
Automa+on
Engineer
does
not
have
QA
experience
Automated
Regression
4
to
6
devices
This
test
suite
will
increase
as
the
new
automated
func+onal
tests
are
completed
for
the
stable
por+ons
of
the
app
Run
a3er
each
build/install/
itera+on
Automa+on
JUnit
(Web
Service
Unit
Tes+ng
and
QA
Framework)
Selenium
(QA
Automa+on)
Appium
(Device
Automa+on)
Gradle
(Build
Tool)
Automa+on
Engineer
Mobile Tes*ng Approach
20

13. 11
Test
Type
Devices
Notes
When
How
Tools
Who
Device
/
OS
Usability
Tes+ng
4
to
6
devices
Make
sure
to
also
test
with
Public
Wi-­‐Fi
End
of
Each
Itera+on
Manual
Test
scenarios
repository,
u+lize
physical
devices
QA
Analysts
End
to
End
Backend
Integra+on
tes+ng
4
to
6
devices
Mobile
App
to
the
Web
Service
to
any
addi+onal
systems
to
complete
the
process
flow
End
of
Each
Itera+on
Manual
Test
scenarios
repository,
u+lize
physical
devices
Business
Testers,
QA
Analysts
End
to
End
/
Func+onal
Acceptance
4
to
6
devices
Run
the
full
func+onal
test
suite
to
verify
that
the
features
func+on
as
required
End
of
Each
Itera+on
Manual
/
Automa+on
Test
scenarios
repository,
u+lize
physical
devices
Business
Testers,
QA
Analysts
Mobile Tes*ng Approach
21
Test
Type
Devices
Notes
When
How
Tools
Who
End
to
End
Device
/
OS
Usability
Tes+ng
10
or
more
devices
Use
actual
devices
in
order
to
get
the
look
and
feel
A3er
App
is
stable
and
prior
to
submikng
to
App
Store
Manual
Test
scenarios
repository,
u+lize
physical
devices
QA
Analysts
Performance-­‐
Page
Load
2
to
4
devices
and
all
Web
App
pgs
Test
real
devices
on
a
variety
of
network
condi+ons
and
bandwidths
(3G,
4G,
Wi-­‐Fi)
every
user
facing
web
applica+on
should
use
Lighthouse
as
a
quality
assurance
gate
against
a
Staging
or
Produc+on
Level
server
Lighthouse
(TBD)
Developers
Mobile Tes*ng Approach
22

14. 12
Test
Type
Devices
Notes
When
How
Tools
Who
ADA
Compliant
NA
Test
a3er
App
is
stable
and
prior
to
submikng
to
App
Store
WAVE
Developers
QA
Security
Penetra+on
-­‐
Dynamic
&
Sta+c
NA
Process
owned
by
Security
Test
prior
to
submikng
to
App
Store
Veracode
Security
Performance
-­‐
Web
Service
load
/
Stress
Test
4
to
6
devices
Concurrent
user
load
test
on
the
Mobile
Web
Service
while
manual
users
are
hikng
the
Mobile
App
Test
a3er
App
is
stable
and
prior
to
submikng
to
App
Store
Automa+on/
Manual
Java
-­‐
TBD
Development
Team,
QA
Analysts
Mobile Tes*ng Approach
23
Test
Type
Devices
Notes
When
How
Tools
Who
Performance
-­‐
Broadband/
Ba[ery
Life
Tests/
Memory
Usage
This
should
be
mi+gated
by
good
architectural
principles
and
applica+on
design
Test
a3er
App
is
stable
and
prior
to
submikng
to
App
Store
Automa+on
TBD
Development
Team
End
User
–
Beta
test
Individuals
devices
Devices
and
OS
limited
to
those
owned
by
Beta
testers.
Ini+al
so3-­‐roll
out
to
a
specific
group
of
users
Test
a3er
App
is
stable
and
prior
to
submikng
to
App
Store
OR
prior
to
so3
roll
out
Manual
Enterprise
App
Store
Selected
group
of
users
with
a
mix
of
age,
loca+on,
etc.
Mobile Tes*ng Approach
24

15. 13
Bug Documenta*on
h[p://adventuresinqa.com/2014/11/06/how-­‐to-­‐file-­‐mobile-­‐bugs/
Opera@ng
System,
Mobile
PlaForm
and
Mobile
Device
• The
same
applies
to
the
opera+ng
system,
the
mobile
plarorm
and
the
mobile
device.
Write
down
the
opera+ng
system,
mobile
plarorm
and
device
on
which
the
bug
occurred.
• Bad:
“On
Android”
or
“On
iOS”
• Good:
“Android,
Version
4.1.2
Google
Nexus
4”
or
“iOS,
Version
6.1
iPhone
4S”
Mobile
Device
Specific
Informa@on
• Mobile
devices
have
lots
of
interfaces
and
sensors
that
could
have
an
impact
on
your
app.
The
ba[ery
could
also
affect
the
app
you’re
tes+ng.
Write
down
all
of
this
informa+on
in
your
bug
report.
• Bad:
“No
informa1on”
• Good:
“GPS
sensor
ac1vated,
changed
the
orienta1on
from
landscape
to
portrait
mode.”
or
“Used
the
device
in
a
sunny
place.”
or
“BaMery
state
was
15%.”
or
“BaMery
state
was
100%.”
We
added
fields
in
our
Defect
tool
for
‘Device’
and
‘OS
version’
which
has
helped
25
Automa*on – Test ID (TID)
Our
Test
ID
Guidelines
for
Developers…
• Assign
ALL
elements
on
a
page
a
TID
not
just
things
that
we
act
on.
Include
elements
such
as
Titles,
Headings
and
Messages
(especially
Error
Messages)
• TIDs
must
be
unique
on
each
page
• Make
them
camel
case
(i.e.
guestPayments
not
guestpayments
or
login
not
Login)
• DO
NOT
CHANGE
them
once
you’ve
checked
in
the
page,
unless
there
is
a
defect
a[ached
to
the
change.
26

16. 14
Miscellaneous
• Created
our
own
Enterprise
App
Store
–
Loca+on
to
get
the
latest
version
of
our
apps
to
test
and
how
we
distribute
apps
to
the
company
that
are
for
internal
use
only
• Once
we
go
to
produc+on
you
can
have
App
Stores
put
in
Beta
version
to
test
“in
Produc+on”
prior
to
having
the
version
release
to
the
public
• Responsive
Web
–
remember
to
not
only
test
the
Web
browsers
but
also
the
different
browsers
on
devices
(Safari,
Chrome,
na+ve
Internet
app)
• Feedback
within
your
app.
Pro
–
not
everyone
will
see
a
bad
review
Con-­‐
you
have
to
have
an
Admin
tool
created
to
gather
the
informa+on
• Do
research
because
everything
will
change
-­‐
tools,
OS,
devices,
etc.
27
Ques*ons
28

17. 15
Further Reading…
• Kno[,
D.
(2015)
Hands-­‐on
Mobile
App
Tes1ng.
New
York,
NY:
Addison-­‐
Wesley.
• Harty,
J.
&
Aymer,
A.
(2015)
The
Mobile
Analy1cs
Playbook:
A
prac1cal
guide
to
beMer
tes1ng.
ISBN
978-­‐0-­‐9970694-­‐1-­‐9
• h[p://www.guru99.com/tes+ng-­‐mobile-­‐apps.html
• h[ps://analy+cs.google.com/analy+cs/academy/
• h[ps://developer.apple.com/ios/human-­‐interface-­‐guidelines/overview/
design-­‐principles/
• h[p://www.belatrixsf.com/whitepapers/mobile-­‐tes+ng-­‐best-­‐prac+ces/
• h[p://istqbexamcer+fica+on.com/mobile-­‐applica+on-­‐development-­‐and-­‐
tes+ng-­‐checklist/
Many
other
materials
out
there…
just
do
a
search!!
29