Five Ways to Improve Your Mobile Testing

T22
Mobile Testing
10/16/2014 3:00:00 PM
Five Ways to Improve Your
Mobile Testing
Presented by:
Dennis Schultz
IBM
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Dennis Schultz
IBM
In IBM’s Emerging Technologies organization, Dennis Schultz is a solution
architect, tester, and software engineer. As a child, Dennis had an insatiable
curiosity of how things worked and a desire to take them apart. The all-too-
common outcome was that he could not put them back together again, thus
leading to an adult career in testing. Wanting to make amends for these
transgressions eventually lead to his current position as a Solution Architect,
helping clients assemble products and tools into integrated solutions to meet
their challenges. Dennis has held roles in IBM from marketing to sales
enablement to technical sales.

© 2014 IBM Corporation
5 Ways to Improve the Quality and Efficiency
of your Mobile Testing
Dennis Schultz, IBM Solution Architect
Rational Emerging Technologies Team
dennis.schultz@us.ibm.com
dennisschultz.wordpress.com
© 2014 IBM Corporation2
Mobile
State of
the Market

Mobile
48% of mobile consumers reported
feeling frustrated and annoyed, with
sites that are not designed with
mobile in mind.
The Mobile Imperative
"Only 16% of consumers
would give a mobile app
more than two attempts.
Poor mobile app
experience is therefore
likely to discourage users
from using an app again."
No Mobile Site = Lost CustomersTwo-thirds of smart phone users say a mobile-
friendly site makes them more likely to buy a
company’s product or service
Mobile internet usagedoubles in growth in theU.S.
By 2014, mobile internet
should take over desktop
internet usage
“By 2014, mobile internet should take over desktop internet usage”
Mobile Shoppers in the US
projected to spend $37.44
billion in 2013, up from$23.72 billion last year
"In 2013 consumers areexpected to spend over$1.25 trillion dollars online,$38.4 billion will come frommobile"
"In 2013 consumers areexpected to spend over $1.25trillion dollars online, $38.4billion will come frommobile""In 2013 consumersare expected to spend over$1.25 trillion dollars online,$38.4 billion will come frommobile“"In 2013 consumersare expected to spend over$1.25 trillion dollars online,$38.4 billion will come frommobile“
"In 2013 consumers areexpected to spend over $1.25trillion dollars online, $38.4billion will come frommobile""In 2013 consumersare expected to spend over$1.25 trillion dollars online,$38.4 billion will come from
Spending on mobile
applications development will
grow by 50% in 2013 to nearly
2% of total IT spend.
"Gartner predicts that mobile AD
projects targeting smartphones and
tablets will outnumber native PC projects
by a ratio of 4:1 by 2015."
"Mobile users will
increase by 91 million,
over the next four
years" (IDC)
“Sales of smart mobile devicesincluding
smartphones and tablets will growby 20%, generate 20% of all IT sales,and drive a whopping 57% of all ITmarket growth"“Sales of smart mobile devicesincluding smart phones andtablets will grow by 20%, generate20% of all IT sales, and drive awhopping 57% of all
market growth“ “Sales of smartmobile devices including smartphones and tablets will grow by20%, generate 20% of all IT sales,and drive
Mobile web growth: 1 in 5
internet users don't use a
computer
Mobile traffic has climbedto more than 16 percentof overall web traffic,increasing 27 percent thisyear.
eMarketer forecasts thenumber of mobile shoppersin the US will increase by24% in 2013 to 118 millionconsumers and represent62% of digital shoppers.
"Since 2009, Mobile Internet usage
has doubled every year"
"A poor mobile experience can have
a negative impact on the
transaction; and in the long term, it
could be even more detrimental to a
brand's reputation"
A Poor Mobile Email Experience Leads 30
percent of Consumer to Unsubscribe
Mobile
Question:
Are the apps you are developing and testing “mission critical”?
Would your management agree?

Mobile
Challenges of Testing Mobile Applications
DEVICE DIVERSITY APP DIVERSITY
ORGANIZATION
DIVERSITY
•  Multiple diverse teams
•  ALM tools
•  Automation tools
•  Business process tester
•  Automation engineers
and programmers
•  Manual testers
MOBILE
PAYMENT
APP
INVESTMENT
APP
MAIN
WEBSITE
INTERNAL
CORPORATE
WEBSITE
INTERNAL
EXPENSE
APP
INTERNAL
TRADING
APPCOMMERCIAL
BANKING
APP
•  Software Platforms
•  Connectivity
•  Hardware
Native Web Hybrid
Mobile
Question:
What type(s) of mobile apps are you developing?
1.  Native
2.  Web
3.  Hybrid

Mobile
Question:
What is your role?
1.  All I do is test and I only test mobile apps.
2.  All I do is test. I test mobile apps, but my day job is testing
other stuff.
3.  I’m a developer and they are forcing me to test what I write.
Mobile
The Impact of Mobile
Development
Testing
Data Management
/ Security
Area / Discipline
PM/Requirements
Management
UI Design Low Large
Low Medium
Huge Huge
Low Low
Process Changes Tool Changes
Low None

Mobile
5 Ways to
Improve the
Quality and
Efficiency
of your
Mobile Testing
Mobile
1.  Implement a collaborative, agile
process using ALM tools
5 Ways to Improve the Quality
and Efficiency of your Mobile
Testing

Mobile
!  Software development involves a team of people
with diverse skills using different tools
–  Design, development, test, operations, etc.
!  Tools can make or break collaboration
–  Fragmented tools create silos
–  Integrated tools break down barriers
!  Effective development depends on …
–  Defined goals
–  Transparency of progress and quality trends
–  Aligning teams across the entire application
development lifecycle
Mobile Software Development is a Team Sport
Mobile
Application Lifecycle Management (ALM) is the discipline of overcoming
organizational silos to realize a whole-team, whole-view approach to the
software delivery cycle.
1. Business
2. Development
Decide
3. Operations*
ALM manages the
flow of:
People
Process
Information
Mobile development
requires the adoption of
Agile practices to keep
up with demand for
frequent releases

Mobile
Question
Do your current mobile development projects use Agile
methods and practices?
1.  Yes, all of them use Agile
2.  No, none of them use Agile
3.  Mix, some of them use Agile
Mobile
What does an Agile paradigm offer?
Agile succeeds three times more
often than non-agile projects
The Chaos Manifesto, Standish Group 2012

Mobile
1.  Implement a collaborative agile
2.  Automate repetitive tests to speed time
to market and improve quality
Testing
Mobile
!  As with software QA, test automation of mobile apps improves the effectiveness
of your QA test process
!  Improve quality
!  Increase device coverage
!  Save money
!  Execute more tests in less time
Automate, Automate, Automate

Mobile
Goals
Test Cases
Frequency
Methodology
Basic acceptance,
build acceptance
Functional testing
across all existing
areas of product
Compatibility
across devices
Very high value
to automate!
High value to
automate
Medium value to
automate
# of Devices
Optimal Test
Strategy
Automated
Testing
Automated
Testing
Manual or
Automated Testing
Smoke
Testing
Regression
Testing
Compatibility
Testing
Deep testing of
new areas of
functionality
Low value to
automate
Manual Testing
New Feature
Testing
Not automated
Exploratory
Testing
Exploration of
functionality from
customer viewpoint
Manual Testing
Efficient and Targeted Automation Strategy
Mobile
Question
Do you use test automation tools today?

Mobile
3.  Adopt a strategy for Continuous
Integration Testing
Testing
Mobile
Service virtualization makes the unavailable become
available for testing
!  Virtual components simulate the behavior
of a service or application during testing
!  System dependencies are a key challenge in
setting up test environments
!  Unavailable/inaccessible services slow down
the testing process
!  Costly 3rd party access fees can significantly
impact testing availability
!  Virtual components run on commodity
hardware, private cloud, public cloud
!  Each developer and tester can easily have
their own test environment
!  Developers and testers can continue to use
current testing procedures and tools
!  Streamline creation of production-like
environments and negative testing
Heterogeneous Environments
Public Cloud
Private Cloud
Data Warehouse Mainframe
Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
App Under TestRouting
Service
Third-party
Services Portals
Content
Providers
EJB
Shared
Services
Archives
Business
Partners
Messaging
Services
Databases Mainframe
applications
App Under Test
Third-party
Services
Packaged apps, messaging services, etc.
Virtual Components

Mobile
Continuous Integration Testing - by example
DevOps Foundation
Continuous deployment and testing across all development stages
Pass/FailApp M1 M2 ERP WSDL 3rd party Database
Real V V V "Test /w back
end systems
Real "Test my
own piece
•  Test Mobile App with three virtualized services.
•  Quick to setup and low-cost.
•  Author integration tests early
#  Service Virtualization is an enabler for continuous integration
testing
#  Services, applications, systems are introduced into the continuous
integration cycle in a prioritized, controlled fashion.
#  Controlled integration helps isolate defects for faster resolution.
Mobile
DevOps Foundation
Pass/FailApp M1 M2 ERP WSDL 3rd party Database
end systems
Real "Test my
own piece
Real V V VR
Real V V VV
Real V V V Test DataVV
Real V V V Test DataVR
$
"
"
"
Integrate
w/another
This won’t
slow me down!
First Complete
System Test
All components
Integrated!
Real V V V Test DataRR "
Test integrations
with control

Mobile
DevOps Foundation
Pass/Fail
Real R R V Test DataRR
Real R R R RRR
"The expensive
backend
All Systems UP!
"
App M1 M2 ERP WSDL 3rd party Database
Real V V VR
Real V V VV
Real V V V Test DataVV
Real V V V Test DataVR
$
"
"
"
Integrate
w/another
This won’t
slow me down!
First Complete
System Test
All components
Integrated!
end systems
Real "Test my
own piece
Real V V V Test DataRR "
Test integrations
with control
•  Accelerate test environment refresh
•  Integrate and test new functionality with control
•  Gradually test integrations with expensive
backend systems
•  Combine continuous test execution with
automated build deployment – Validate not only
the code being released but also the
deployment process
•  Discover and isolate defects faster!
•  Accelerate delivery of higher quality software!
Mobile
Question
Are you using service virtualization or
continuous integration testing today?

Mobile
Integration Testing
4.  Utilize a combination of real and
emulated devices
Testing
Mobile
Different screen
resolution /
screen size
Unreadable text, blurred images,
misalignment of screen elements,
and items that fall off the screen.
Android
Customizations
On-screen and physical controls
that function differently across devices
Customized handling of inputs and
events (e.g. Samsung Swype vs.
default Android keyboard).
!
Memory / CPU Low or insufficient memory or
processing power. Impacts of other
services running on the device.
Mobile device characteristics that impact the quality of your application or website, but
are not easily verifiable using an emulated phone or browser, include:
Why Test on Real Devices?

Mobile
!  Device platforms, fragmentation, and growth
•  iOS, Android, BlackBerry, Windows Phone,
Symbian, etc.
•  Multiple form factors and screen resolutions
•  Device refresh every 24 months
•  In 2011, over 300 new devices released
!  New capabilities to test
•  Camera, GPS, direction, orientation, voice, etc.
!  More network considerations
•  Multiple carriers, variable throughput and latency,
disconnected use, network switching, etc.
Testing on Real Devices is Challenging
Mobile
Question
How much of your mobile testing is done using real
devices?
1.  0 – 1/3
2.  1/3 – 2/3
3.  2/3 – All of it

Mobile
Smoke
Testing
Unit
Testing
Mobile Product Development Timeline
TestPhase
Emulators Real Devices/ Device Cloud
Emulators Real Devices/ Device Cloud
New
Feature
Testing
Emulators Real Devices
When to use Emulators/Simulators vs Real Devices
Dev QA
OS&
Compatibility
Testing
Device Cloud
Mobile
Real Devices in the cloudYour computer, connected
to the internet
Your key presses and
mouse clicks sent to
the device
The device’s screen
sent back to your
computer
SaaS-based Remote Device Products

Mobile
Devices
Shared Devices
!  24X7 access to hundreds of smart
devices
!  Use for compatibility testing,
“untrusted device” testing
Private Devices
!  24X7 access to your devices
!  Enables geographically diverse
team
!  Inside corporate firewall or external
Local Devices
!  Plug your smart devices on hand
directly into your local computer
!  Make use of existing assets, great
for local teams
Use a Combination of Local and Remote Devices
Mobile
Integration Testing
emulated devices
5.  Focus on the user’s experience
Testing

Mobile
Concerns that make Mobile App Quality different
Tester
End UsersDeveloperLOB/Digital Marketer
When someone leaves a negative review in the app store, it scars your
app for life, you can’t respond to it, and you can’t learn more about the
problem in order to fix it quickly.
- Mobile Orchard
•  How effective is our mobile
app at engaging
customers?
•  How can I stay on top of
current quality metrics, and
turn them into an
actionable strategy?
•  How could I get visibility to
issues that customers
experience in production
so that I can address them
quickly?
•  How can I get more
people to test the apps so
that we can cover different
devices, network carriers,
and OS versions?
•  How can I get the latest
and greatest build without
wasting my time to track
that down?
•  How can I quickly submit
bug reports with
screenshots and device
details right from the
application?
•  How can I provide
direct and
constructive
feedback?
Mobile
Tester
End UsersDeveloper
Over the air build distribution1 In app bug reporting2
Crash log reporting
3
In app user feedback4
LOB/Digital Marketer
Sentiment Analysis5
Builds
User Feedback
Crash logs
Bugs
Bugs vs. Crashes
IBM Mobile Quality Assurance
Continuously deliver high quality mobile apps

Mobile
Integration Testing
emulated devices
5.  Focus on the user’s experience
Testing
Mobile

Mobile
© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind,
express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have
the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM
software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities
referenced in these materials may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature
availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines
Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

Five Ways to Improve Your Mobile Testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (18)

Similar to Five Ways to Improve Your Mobile Testing

Similar to Five Ways to Improve Your Mobile Testing (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Five Ways to Improve Your Mobile Testing