Mobile Testing: Challenges and Solutions

6/7/17
1
Mobile Tes*ng Challenges and
Solu*ons
@philiplew
@xboso7
© 2017 XBOSo7, Inc.- All Rights Reserved. 1
XBOSo7, Inc. All Rights Reserved.
Expecta*ons
1. I won’t read the slides
2. Lots of material
3. You can have it, just email me
a. philip.lew@xboso7.com
2

6/7/17
2
Tiny Bit About Me
• Phil Lew
– Telecommunications consultant and network designer
– Team Lead, Data warehousing product development
– Software product manager, BI product
– COO, large IT services company
– CEO, XBOSoft, software qa and testing services
• Relevant specialties/Research
– Software quality process improvement
– Software usability evaluation
– Software quality in use / UX design
3
But Today is All About You
… Who are you and Why are you here?
4

6/7/17
3
Agenda
1. Mobile Vs. Desktop Platforms
–  Fundamental differences between the mobile and desktop
platforms
–  Foundation to further explore mobile software best
practices.
2. Challenges in Designing and Testing Mobile
Applications with solution approaches
–  Specific design issues associated with the creation of a
great user experience (UX)
–  Localization
–  Testing and automation considerations inherent to
complex, mobile devices.

5
Mobile vs. Desktop PlaVorms
1.1 Similari*es
• Leverage PlaVorm Development
•  Across web interfaces (HTML/Responsive
Web Design) … when you can…
•  Across data connections of an application
across different platforms. using APIs …
when you can…
•  Across similar code bases for a given
application on different platforms, helped by
the IDE (e.g., Xcode).
6

6/7/17
4
Mobile Vs. Desktop PlaVorm
1.2 Differences
… Leveraging similari5es provides simple
solu5on, but...
• Tendency to accept similari*es
– Fails to address subtle UX requirements
•  Inherent/plaVorm-specific peculiari*es
– Func*onal/applica*on-specific
… Testers need to be aware of these differences
7
Mobile Vs. Desktop PlaVorm
1.2 Differences
8
1. PlaVorm prolifera*on
2. Loca*on
3. Photography
4. No*fica*ons
5. Content
6. Research (search) tasks
7. Chat
8. Email
What are the top
tasks that you
use your mobile
phone for?

6/7/17
5
Mobile Vs. Desktop PlaVorm Diﬀerences
1.2.1 PlaVorm Prolifera*on

"Android Fragmentation Visualized (August 2015) - SlideShare." 7 Aug. 2015,
http://www.slideshare.net/FrdricZugaj/android-fragmentation-visualized-august-2015.
9
• Over 18000 Unique Devices Detected
• Android a Far Bigger Problem Than iOS
• Fragmenta*on a Strength and Weakness for
Android*
– Headache for developers and testers
– Access to large variety of devices oﬀers broad,
global reach
* "ANDROID FRAGMENTATION VISUALIZED (AUGUST 2015) DEVICE ...."
https://opensignal.com/assets/pdf/reports/2015_08_fragmentation_report.pdf.
1.2.1 PlaVorm Prolifera*on (cont.)
10

6/7/17
6
Screen Size/Aspect Ra*o Varia*on Challenges
•  Large Numbers of Android Displays (le7)
•  Rela*vely Small Variety of iOS Displays (right)
•  Developers Forced to Priori*ze
•  Testers Need to Accommodate

1.2.1 PlaVorm Prolifera*on (cont.)
11
• Mobile Environment
– Mobiles accessed 150 *mes a day
– Mobiles constantly changing loca*on and
condi*ons
• Loca*on Based Services
– Unique “mobile-only” loca*on-based services
– Added Services Dimension
•  Naviga*on, restaurant reviews, etc.
• Ambient Condi*ons
– Adjust to varying ligh*ng condi*ons

1.2.2 Loca*on
12
What other conditions
must be accounted for?

6/7/17
7
•  Enabling
–  Mobile cameras always with you
– No real desktop equivalent
•  Mobile Cameras U*lized 10x Over Tradi*onal
Cameras
–  Scanning as well

1.2.3 Photography
13
1.2.3 Photography (cont.)
14

6/7/17
8
•  Applica*ons - New Social Paradigm
–  Rapidly take pictures
–  Quick edi*ng (e.g., Instagram and Facebook
Filters) and sharing
• Requirements for UI to be Supported
– Rapid access to take photos
– Rapid storage
– Edi*ng photos must be seamless and simple

1.2.3 Photography (cont.)
15
• No*fica*on Implementa*on Approach
– Directly affects overall UX
• Poten*ally Large Number of No*fica*ons
Received by Mobile
– And desktops associated with mobile
– Receipt of no*fica*ons at inappropriate *mes
and loca*ons
– Give users a choice that’s easy to find and
change

1.2.4 No*fica*ons
16

6/7/17
9
… Mobile Devices - Always with you
• Ideal for Diges*ng Content During Idle Times
• UI Should be Designed for Fast Access
– During short *me availability (e.g., in line at
grocery store)
– Large bunons and gestures
• Efficient Power Design
– For both device and applica*on

1.2.5 Content
17
• Limita*ons
– Small Screen sizes
– Input Difficult
• Benefits and Customiza*ons
– Loca*on component
– Applica*on UI
•  Minimize screen input
•  Intelligent, automa*c access of loca*on info
WITHOUT user interven*on, allow choice

1.2.6 Mobile-Based Research (searching)
18

6/7/17
10
• Limita*ons
– Input limita*ons
– Always with you
– Ecosystem of apps can exist within Chat app
•  Mini Apps (e.g., WeChat, Facebook)
– Applica*on UI
•  Simple and minimal
•  Use of canned responses
•  Short, mul*ple interac*ons on the go

1.2.7 Chat Applica*ons
19
• Limita*ons
– Input methods
– Viewing screen size
•  rendering comprises
– Reduced feature set
– Always with you
– Customized UI
•  Autocorrect has drama*cally improved, reducing
fa*gue and increasing accuracy
•  Use of canned/customized responses

1.2.8 Email
20

6/7/17
11
2. Challenges in Designing and
Tes*ng Mobile Apps
2.1 UX Cri*cality
2.2 Localiza*on
2.3 Tes*ng and Automa*on

21
Challenges in Designing and Tes*ng Mobile Apps
2.1 User Experience Cri*cality
•  UX Essen*al to Mobile Applica*on
•  Development is a Team Eﬀort
•  Design -> Coding - > Tes*ng
•  Not Limited to Discrete Func*onal Capability
•  Must Meet Demands of User Expecta*ons for
Overall UX
•  Maintain Key Tenets of Brand’s Value

22

6/7/17
12
•  Tradi*onal Business Perspec*ve Applies…
–  It is cheaper and less *me-consuming to keep
your current customers than acquire new ones
•  … and Is Ampliﬁed
–  Level of compe**on for Applica*ons greater
–  App Stores simplify ﬁnding alternate solu*ons
–  Opinions can be formed in seconds

2.1 User Experience Cri*cality (cont.)
23
• Localy*cs:
–  25% of Applica*ons will be used ONCE
–  33% of applica*ons fully abandoned a7er 1
month
–  70-80% Fully Abandoned a7er 3 months
• Dimensional Research
–  80% of app users will anempt to use a
problema*c applica*on less than 4 *mes

"Mobile Apps: What's A Good Retention Rate? | Localytics." 21 Mar. 2017,
http://info.localytics.com/blog/mobile-apps-whats-a-good-retention-rate.
"Whitepaper: Failing to meet mobile app user expectations | ITProPortal (REF: Dimensional Research) ." 30 Jan. 2016,
http://www.itproportal.com/2016/01/30/whitepaper-failing-meet-mobile-app-user-expectations/. Accessed 21 Apr. 2017.
24

6/7/17
13
25
86% didn’t
use after 3
months
UX Dependencies
•  UI
–  First thing new user will see
–  Responsive and Purpose-Built
–  Provide expected results with few obstacles
–  No confusion in naviga*on or understanding
•  Annoying User Experiences Lead to Abandonment
–  …. “Simple but Mediocre” will trump “Comprehensive but
Diﬃcult”

26

6/7/17
14
… Data Speed Challenges for Mobile Applica5ons
•  Operate in Widely Varying Wireless Environments.
•  Deal Gracefully With Interminent Data Connec*ons
–  Intelligent Caching
–  Moving between wireless interfaces (WiFi, Cellular,
Bluetooth)
•  UI Op*mized to Maximize UX (... Illusion)
–  Slow download speeds masked with progress screens
–  Inform User of progress
–  Distract User from Slow progress

User Experience Cri*cality
2.1.1 Performance Issues
27
•  Overall Performance Measurements and Monitoring
–  Data Throughput
–  Screen Delays
–  Banery Consump*on
–  Database Performance
–  Memory U*liza*on (leaks, abandoned memory)
–  Automated Source Code Issues review
•  Mobile Performance Tool Examples
–  XCode’s UI Tes*ng Infrastructure (previously
Instruments) (Apple Inc.): iOS
–  Monkop : Android, iOS

2.1.1 Performance Issues (cont.)
28

6/7/17
15
Mobile Device Unique
•  Always with the person
•  Typically held in one hand
•  Biometrics availability
… More a personalized extension of user, less an
independent device
How many of you would loan your phone to a friend?

2.1.2 Design Controls Based on Hand Posi*on
29

… 50% of People Hold Phone in 1 hand, using thumb
to drive interface. *

… 75% Interac5ons on mobile are thumb driven. **

* "Design for Fingers, Touch, and People, Part 1 :: UXmatters." 6 Mar. 2017,
http://www.uxmatters.com/mt/archives/2017/03/design-for-fingers-touch-and-people-part-1.php. Accessed 19 Apr. 2017.
** "How We Hold Our Gadgets · An A List Apart Article." 3 Nov. 2015, https://alistapart.com/article/how-we-hold-our-gadgets.
Accessed 19 Apr. 2017.
2.1.2 Design Controls Based on Hand Posi*on
(con*nued)
30

6/7/17
16
Thumb-zone mapping for left- and right-handed users. *
* "The Thumb Zone: Designing For Mobile Users – Smashing Magazine." 19 Sep. 2016,
https://www.smashingmagazine.com/2016/09/the-thumb-zone-designing-for-mobile-users/.
2.1.2 - Design Controls Based on Hand Posi*on
(con*nued)
31
•  UX Formulated by So7ware and Hardware,
Together
•  Physical Home Bunon Easily Accessible
•  Virtual Bunons and Controls Easily Reachable
•  Hap*c Feedback Helping Blur Lines Between
Virtual and Real
… Mobile device Ease of Access and Use by a phone
naturally placed in one’s hand will deﬁne the UX
2.1.2 - Design Controls Based on Hand Posi*on
(con*nued)
32

6/7/17
17
•  Less Deﬁned Interac*ons
– Thumb Driven inherently less accurate
– Swiping gestures, pressing Icons
•  Text Input
– Main challenge for “small gesture” requirement
– Capaci*ve Screens w/correc*on algorithms
– S*ll imperfect and tedious
•  Should be kept to a minimum
•  Employ autocomplete for forms

2.1.3 - Design to Larger Gestures

33
Text Size
• Do NOT Maximize Informa*on On Small
Screen
• Text Should Be Easily Readable
– in Mobile-Moving Environments
•  e.g., on a Bus, in a car
– When Displaying User-facing screen elements
•  e.g. table column headings, bunons, help menus

2.1.4 - Design For Visual Clarity

34

6/7/17
18
Contrast
•  Flexibility in Contrast Can Be
Aesthe*cally Pleasing
… however…
•  Users With Marginal Vision
Should NOT Have to Struggle
•  Contrast Setng Guidelines
Available From www.w3.Org
"The Logical Blog by IconLogic: eLearning : Three Tips For Better Color ...." 7 Mar. 2017,
http://iconlogic.blogs.com/weblog/2017/03/elearning-three-tips-for-better-color-usage.html.
2.1.4 - Design For Visual Clarity
(con*nued)

35
Use Complementary Colors
To Create Contrast
© 2017 XBOSo7, Inc.- All Rights Reserved. 36
By Kwamikagami - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=43056011
Do you see
your favorite
sports team
colors here?

6/7/17
19
• Task-Driven
–  Clear view to primary ac*on
–  Secondary ac*ons should be added, but only if
necessary
• All Ac*ons Should Be Intui*ve
–  Easy to learn with nothing to memorize
–  Easier to use
–  Easier when expanding func*onality

2.1.5 - Design for Simplicity

37
Fidelity
•  Easy to see
•  Clear objective
– Simple
– Not everything
38 © 2017 XBOSo7, Inc.- All Rights Reserved.

6/7/17
20
Design to leverage Context-Aware Sensors
•  Expanding Prolifera*on
•  Development Toolkits Support Coding and Easy
Access
•  Eliminate Need for Users to Manually Input Data
–  e.g., GPS Applica*on, where User’s loca*on immediately
known
2.1.5 - Design for Simplicity
(con*nued)

39
What other sensors are becoming popular?
"OpenSignal Mobile Sensors - OpenSignal." https://opensignal.com/sensors/library/.
2.1.5 - Design for Simplicity (con*nued)

40

6/7/17
21
•  All Ac*ons Should Be Intui*ve if Possible
–  Easier to learn with no memoriza*on
–  Easier to use
–  Easier when expanding func*onality.
–  Eliminate the need for Reference/Help Files
•  U*lize Well-Established Norms for the Mobile OS
–  iOS: Human Interface Guidelines (Apple)*
–  Android: Design | Android Developers”**

* "Design Principles - Overview - iOS Human Interface Guidelines."
https://developer.apple.com/ios/human-interface-guidelines/.
** "Design | Android Developers." https://developer.android.com/design/index.html. Accessed 10 Apr.
2017.
2.1.6 Self-Evident Naviga*on

41
•  Development For Mul*ple-PlaVorms
•  User Expecta*ons
–  Func*ons will be duplicated or complemented on
desktop and mobile applica*on
•  Design Considera*ons
–  UX principles be duplicated where possible
–  Aggressively modify func*on to support mobile UI where
appropriate
–  Developer should abandon desktop func*ons
inappropriate for Mobile environment
2.1.7 - Seamless Experience Across Devices

42
What functions would be inappropriate?

6/7/17
22
Spotify Application on Various Devices
2.1.7 - Seamless and Con*guous Experience
Across Devices

43
•  Manual Tes*ng
–  Simple applica*ons can test for look and feel
–  Usability
•  Serious Applica*ons
–  So7ware Tes*ng QA Team integral part of SDLC
–  Manual tes*ng cannot be avoided for many areas
–  With automa*on of certain func*onal areas
2.1.8 - UX Tes*ng Challenges

44

6/7/17
23
Manual UX Review
•  Access Right-Hand Versus Le7-Hand Usability
•  Screen Interac*on (Small Items)
•  Visual Clarity
–  i.e., contrast of displayed informa*on under various
ligh*ng condi*ons
•  General Naviga*on Through Applica*on
–  Adherence to appropriate device guidelines
–  Conﬁrma*on of cross-plaVorm uniﬁed experience
2.1.8 - UX Tes*ng Challenges (cont.)

45

UX Evalua*ons Via Manual Methods
•  Well planned tests
•  Hallway tests
•  Coordina*on with other SDLC Stakeholders

2.1.8 - UX Tes*ng Challenges (cont.)

46

6/7/17
24
Set Up a Simple UX Lab for $200
Note Taker
Observer
Facilitator
User
Executing tasks
47
Work in small groups of 2-4
1.  Access the Costco (or United airlines) website
on your mobile device
2.  Access the Costco (or United airlines) website
on your laptop
3.  Iden*fy the diﬀerences if any and why or why
not?

Mobile Tes*ng: Challenges and Solu*ons
UI Challenge 1
48

6/7/17
25
• Localiza*on Challenges
–  Google Play Store, iOS App Store
–  Distribu*on of Applica*on Globally is easy
–  Crea*ng Suitable Applica*on for a Market is hard
• Applica*on Market Acceptance
–  Local Language
–  Market-speciﬁc Images

2.2 - Localiza*on
49
… 90% of all mobile ac5vity takes place in Apps,
not browsers
… App Localiza5on will increase downloads, on
average well over 100% *

"There's a Language for That: Translating Mobile Apps And Content."
http://www.demandgenreport.com/features/demanding-views/there-s-a-language-for-that-translating-
mobile-apps-and-content.
2.2 - Localiza*on (cont.)
Price for Not Localizing
Who is your market?
50

6/7/17
26
Cri*cal Localiza*on Considera*ons
•  Assets
–  images, tutorials, other content
•  Layout Flexibility - Ensure That Text Will Autosize
–  Chinese requires less space than English
–  Spanish, French, German take up more space than
English
–  iOS/Android Toolkits help with dates, *mes, currency

51
Cri*cal Localiza*on Considera*ons (cont.)
Transla*ons
•  What to localize?
–  En*re applica*on
–  Help ﬁles, menus, other references
–  L1, L2?
•  Na*ve transla*ons by subject maner experts
–  Must understand context

52

6/7/17
27
Tes*ng and Review of Localiza*on Data
•  Reviewer
–  Must know the language
–  Must know subject and context
•  Compa*bility Tes*ng
–  Accurate rendering
•  Use of device pools to test applica*on on target devices
–  e.g., Sauce Labs, BrowserStack, AWS Device Farm
–  Automa*on of compa*bility tests (covered in next sec*on)
53
2.3 Tes*ng and Automa*on
1.  Wireless interfaces
2.  API integra*on
3.  Test tools
4.  Device diversity
5.  What to automate
6.  Mobile test lab
7.  Con*nuous integra*on
8.  Mobile data security
54

6/7/17
28
•  Cap Gemini Survey from 2014
–  20% of companies have no *me to test mobile apps
–  65% lacked appropriate tools
–  50% lacked devices to perform tes*ng
•  Mobile Tes*ng Landscape is Changing
–  Different applica*ons, be they web apps or na*ve,
require different tes*ng tools and approaches
–  Numerous device interfaces that need to be supported

Challenges in Designing and
Tes*ng Mobile Apps
55
Mobile Devices Operate in Complex Environment
•  Data Feeding From a Variety of Services
–  Wireless - High Speed
•  Indoors: Wifi
•  Outdoors: Cellular (e.g., 3G, 4G, 5G)
•  Device Moves Automa*cally Between Services
–  behind the scenes
•  Data Handling
–  defaults set to avoid cellular service fees if possible.
–  handling of data interrup*ons

Tes*ng and Automa*on
2.3.1 Wireless Interfaces
56

6/7/17
29
API Considera*ons
•  API Integra*on to External Services Exploding
•  Widespread Use
•  Prone to Errors and customized
implementa*ons

2.3.2 API Integra*on
57
Underes*ma*ng of Tes*ng Challenges
•  Some APIs Are Well Established (Less Risk)
–  e.g., music services
•  Many APIs Unstable
–  More eclec*c sets of devices (e.g., IoT) feeding diﬀerent
informa*on to mobile applica*ons
–  Immature API development
•  Care Must Be Taken
–  To properly evaluate and test less Mature APIs
–  Coopera*on with API service provider

2.3.2 API Integra*on (cont.)
58

6/7/17
30
Tool Selec*on
•  Many Things to Consider
–  Cost (Open Source/Free vs. Paid)
–  OS coverage (e.g., Android)
•  Cost Is a Factor, but Not the Most Important
–  Will the applica*on do the job?
–  Is the applica*on supported (na*ve vs. web)
2.3.3 Tes*ng Tools
59
Mobile Test Automation Tool Comparison
"5 Mobile Testing Challenges and Solutions — Software Testing Help."
http://www.softwaretestinghelp.com/5-mobile-testing-challenges-and-solutions/.
Tool
Paid/Open
Source
Native Apps Web Hybrid Apps Android IOS Windows Black-berry Library/Tool
Robotium
Open
Source
Y - Y Y - - - Library
Sikuli
Open
Source
Image Based
Image
Based
Image
Based
Y Y Y Y Both
Selenium WebDriver
Open
Source
- Y - Y
Y(but
obsolete)
- - Library
NativeDriver
Open
Source
Y - - Y Y - - Library
Appium
Open
Source
Y - Y Y Y - - Tool
MonkeyTalk
Open
Source
Y Y Y Y Y - - Tool
SeeTest Paid Y Y Y Y Y Y Y Tool
M-eux(Jamosolutions) Paid Y - Y Y Y Y Y Tool
EggPlant Paid Image Based
Image
Based
Image
Based
Y Y Y Y Tool
mAutomate Paid Y Y Y Y Y - - Web Based
Ranorex Paid Y Y Y Y Y - - Tool
60

6/7/17
31
XBOSo7 Preference
•  Standardized on Appium
–  XBOSo] holds no Stake in Appium!
–  Appium is open source
•  Answers two ques*ons for us
–  Does the Job of Tes*ng Na*ve Mobile Applica*ons
–  Supports the Applica*ons (iOS and Android)

2.3.3 Tes*ng Tools (cont.)
61
Appium Highlights
•  Appium uses Vendor-Provided automa*on
frameworks
–  XCITest for iOS
–  UIAutoma*on for Android,
–  WinApp Driver for Windows
•  No Appium or 3rd Party code compiled into App
–  Tes*ng same app being shipped

62

6/7/17
32
Appium Highlights (cont.)
•  Appium wraps Vendor Frameworks in the WebDriver API
(aka “Selenium WebDriver”) that speciﬁes a client-server
protocol (JSON Wire Protocol)
•  With Appium, testers can use variety of frameworks and
programming languages
•  WebDriver has been come a standard for test automa*on
with Web browsers, it is not necessary to do anything
diﬀerent for mobile.
•  Open Source
–  Plenty of support from ac*ve developer community.
63
Appium Supports the Following Test Engineering
Requirements*
•  Commonly Used Programming API's Can Be Integrated
•  Can Run App Through Appium Server Without Manipula*ng
the App
•  No Source Code Required (Unlike Robo*um)
•  CI Compa*ble With Jenkins
•  Runs on Selenium Grid
•  Works With Real Devices or Simulators, or Even Devices
Provided by Cloud Services
•  Doesn't Require an APK
"Advantages and Disadvantages of appium - Stack Overflow." 6 Feb. 2015,
http://stackoverflow.com/questions/28363221/advantages-and-disadvantages-of-appium/28367587.
64

6/7/17
33
•  Too Many Device Varie*es to Test
•  Employment of Analy*cs
–  Help Priori*ze Device Focus
•  Example Parameters for Market Data Collec*on
–  Screen Size, OS, CPU, RAM Usage, etc
•  Analy*cs Helps Designers Priori*ze Device Models
–  Helps determine type of support an app will provide *
•  Test Engineers Need to Ensure
–  Priori*zed device conﬁgura*ons appropriately supported
* "Five types of mobile analytics data and their uses." 10 Jun. 2016,
http://mobilebusinessinsights.com/2016/06/five-types-of-mobile-analytics-data-and-their-uses/.
2.3.4 Device Diversity
65
•  Manual tes*ng is essen*al to evaluate overall UX
•  Look and Feel
•  Baselines for test are established

Automa*on
•  Necessary in today’s business environment
•  Challenge is what to automate and how to priori*ze

2.3.5 What to Automate
66

6/7/17
34
•  Some Things Should NOT Be Automated
•  Some Things CAN Be Automated and make
sense to do so:
–  Certain established look-and-feel tests that are
repeatedly successful
–  Asser*ons can be applied to ensure tests are within
bounded parameters.

2.3.5 What to Automate (cont.)
67
Based on that, these types of test cases are
good to be automated:
•  Have been successfully executed manually
•  Stable areas, pages that are not likely to
change frequently
•  Transac*ons, workﬂows that are not likely to
change

Good to Automate
68

6/7/17
35
Difficult to Automate
These test cases, due to the differences in mobile, are
difficult to be automated, the cost of both automa*on
and maintenance would be high:
•  New features without manual tes*ng
•  Complex gestures
•  With no known oracle (expected or correct result/
behavior)
•  Switch across Apps, Interrupts
•  Dependences like SIM card, camera, etc

69
Test Automa*on Script Guidelines
Similar to all automa*on tes*ng, we want the
automated script for mobile to be
•  Small - Easier to understand and fix
•  Fast - Parallel execu*on for faster feedback
•  Independent - Can run any subsets in any order
•  Repeatable - Tests get the same result every *me
•  Self-Checking - No human checking
•  Reusable - Avoid maintenance nightmares
70

6/7/17
36
•  Local and Remote Cloud Test Lab Conﬁgura*ons
–  Both Have Advantages
•  Suitable When Support Large Number of
Devices and OS Versions Is Required
–  Accomplished by various means:
•  Local Real Devices
•  Local Simulated Devices
•  Remote Real Devices
•  Remote Simulated Devices

2.3.6 Mobile Test Lab
71
Local Device Simulator
•  Virtual device
–  not actual phone
•  Phone emulated through so7ware
•  Same OS and so7ware (in large part) as that
loaded on actual mobile hardware
•  Performance may vary from real device
–  CPU/RAM/Network Speed
•  Typically free (for personal use)

2.3.6 Mobile Test Lab (cont.)
72

6/7/17
37
XBOSo7, Inc. All Rights Reserved. 73
Tes*ng and
Automa*on
(cont.)
Examples of local
Android simulator
https://
www.genymotion.com/
(android only)
Examples of local iOS simulators
(Apple XCode Built-in Simulator list)

6/7/17
38
Typical Local Software Test Lab
75
Remote Device Farm Server
•  Similar to Local So7ware Test Environment
•  Majority of Local Test Conﬁgura*on Func*on
Moved to Cloud
•  Local Tester Does Not Have to Physically
Manage
–  Remote test client so7ware
–  Growing Device Pool

76

6/7/17
39
Typical Remote Software Test Environment
77
Remote “Real” devices
•  Typically Most Recently Available Mobile Devices
•  Devices Accessed Through Web Browser
–  not actually holding phone
–  “mirrored image” of real device

78

6/7/17
40
Cloud-based
Real iPhone
Mirrored in
Local Firefox
(BrowserStack)
79
What you can
expect when
using
simulators
locally, or in
the cloud, or if
using real
devices in the
cloud.
Remote
Simulated
Device
Mirrored in
Local
Chrome
(AWS Device Farm)
80

6/7/17
41
Simulated iPhone running on remote MacOS Desktop
viewed on local Chrome Browser (BrowserStack)
81
Available Devices - BrowserStack Example
82

6/7/17
42
Expected Result

(cont.)
83
Local Simulated Devices in Local Mobile Test Lab
•  Advantages
–  Supplements real device pool
–  Large array of devices available
–  Can run automated tests simultaneously on mul*ple
devices (e.g., Using Appium)
•  Issues
–  Limited Look-and-Feel automated test applicability
–  Requires some maintenance to upkeep lab
–  Limited to local computer power

84

6/7/17
43
Remote Test Lab (Device Farm)
•  Advantages
–  Large assortment of real and simulated devices
–  One URL can be displayed against numerous browsers
–  No maintenance required for local devices
–  Automated Tests are Supported
•  Issues
–  Automated Tests can get costly if required to run
simultaneously
–  Limited Look-and-Feel automated test applicability
85
Real/Simulator/Cloud Device
When is best to use which one (1/3)
86
Situation Solution Why
During the initial stages of
application development /
coding phase
Simulator Fast debugging
Sanity, smoke testing,
performance,
interoperability and
network feasibility and
regression testing
Real
Device
A series of real devices with high
popularity is always the final gate, real
usage with real situation must be
covered before final release.
Web application through
URL
Simulator It’s much easier to copy url to
simulator than input it to real phone
Situation-based
application
Real
Device
Only real device can answer: is it easy
to use the app on the train, or while
walking down the street? The situation
about in bright sunlight or in the rain?

6/7/17
44
Feeling of closeness
towards the real
handheld devices
Real
Device
Look and feel, color resolution of the screen,
whether the picture is bright or not under both day
and night conditions and so on
Capturing
screenshots of the
situations where
defects appear
Simulator Capturing issue of screenshots over simulator is
much easier for manual testing. For automated
testing, screen captures can be configured for
simulators as well as real devices.
Battery/exact color/
incoming interrupts/
Camera / Dynamic
lighting conditions
Real
Device
Simulators are not able to simulate the Battery
status, or the exact color display of the devices
when the real device is in sunlight or in the dark, or
simulating incoming interrupts for SMS or calls and,
no cameras.
Performance Real
Device
The original devices tend to perform faster than the
emulator/simulator.
Memory related
issues
Real
Device
The memory available at the simulator tends to be
far more than the real devices so this may create
misconceptions
Responsive
Design Testing
across Devices
Cloud
Device
No hardware factors as what matters are just width and
ratio, Cloud device farm provide all that.
Cross Browser
Testing
Cloud
Device
Safari, Chrome, IE, Firefox, different version, on
different OS, with different screen size, the cloud service
is designed to manage all of it.
Automated
regression test
script
Cloud
Device
Make sure the script works for 1 real devices locally,
and then have the cloud to multiply it to 20 or even
more, in parallel, and you don’t have to maintain any of
them.
Devices with
lower popularity
Cloud
Device
Cover the devices in your supported list as much as
possible.
Call center,
customer
reported issues
Simulator Troubleshooting the issues with simulators first before a
real device is available.

6/7/17
45
Automated Mobile Applica*on Tes*ng
… Periodically Scheduled
•  Appropriate Tests Are Created
– Various func*onal areas
– Projects run in parallel to save *me
•  Tests Are Scheduled and Run

2.3.7 Con*nuous Integra*on
89
Con*nuous Integra*on
•  Kicked Oﬀ Immediately A7er Latest Applica*on
Build Completed
•  Typically Runs Smoke Tests
–  To exercise cri*cal test areas for proper conﬁgura*on
and opera*on
–  Done prior to detailed test run
•  Local Test Environment
•  Database Ini*a*on and Access
•  Network and device access
•  General Applica*on tes*ng of major func*onal areas

2.3.7 Con*nuous Integra*on (cont.)
90

6/7/17
46
Con*nuous Integra*on Beneﬁts
•  Well suited for Agile Development teams
•  Can Ini*ate Test Process Directly A7er Latest
Applica*on Build
•  Rapid Feedback
–  rather than wai*ng un*l next day
•  Iden*ﬁed Problems Can Be Quickly Addressed

2.3.7 Con*nuous Integra*on (cont.)
91
Mobile Data Security Considera*ons
•  Extremely Important, Specialized
•  Open-Source as Well as Commercial Tools
Available
– OWASP (Open Web Applica5on Security Project)
•  Certain Security Tests can be Performed by
Performance Tools
– e.g., JMeter DDOS, Monkop

2.3.8 Mobile Data Security
NOTE: Due to Depth and Nature Mobile software security,
XBOSoft covers this topic in a separate presentation.
92

6/7/17
47
Summary (1/3)
Diﬀeren*a*on of Requirements for Desktop/
Notebook Computers Versus Those for Mobile
Devices
•  Design Parallels should be leveraged wherever
possible
•  Cau*on using *me-saving techniques
–  e.g., web apps to gain broad plaVorm footprint rather
than device plaVorm speciﬁc na*ve applica*ons that
op*mize UX.

93
The Needs of the End-User of the Applica*on Must
Served, First and Foremost
•  Which devices to be supported
•  Which sensors to be accessed
•  Where app will be used
•  Can the applica*on easily accessed and *mely in data
presenta*on
•  How will the user be holding the device (which hand)?
•  Can Elegant handling of data through mul*ple data services
be supported? (e.g., 4G vs. WiFi)

Summary (2/3)
94

6/7/17
48
•  Mobile Design considera*ons are
Complex
•  With More Mobile devices, plotng out
your test strategy is cri*cal
–  Resource limita*ons are ampliﬁed
•  Time and Money (test environments)
–  Leverage test automa*on
–  Leverage ‘combo’ lab depending on needs

Summary (3/3)
95
Ques*ons and Answers
Philip Lew
@philiplew
philip.lew@xbosoft.com
Some resources: http://www.xbosoft.com/resources
© 2017 XBOSo7, Inc.- All Rights Reserved.
www.xbosoft.com
@philiplew @xboso7
Mobile Tes*ng
Challenges and Solu*ons
96

6/7/17
49
•  Find an Applica*on that you consider requires a
redesign due to UI and/or UX issues.
•  Write down the issues you are seeing
•  Write down any improvements you believe can
be made to improve the applica*on
•  If possible, be ready to demonstrate the issues
live on the phone.

Prac*ce Exercise 1
97
•  For a project you have recently or are currently
working on, explain some of the issues that
created the biggest UX challenge
–  As a Developer, or
–  Project Manager working with a so7ware
team

Prac*ce Exercise 2
98

Mobile Testing: Challenges and Solutions

Recommended

Recommended

More Related Content

Similar to Mobile Testing: Challenges and Solutions

Similar to Mobile Testing: Challenges and Solutions (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Mobile Testing: Challenges and Solutions