Are you applying a systematic approach that provides sufficient test coverage of devices, operating systems and network operators? Our whitepaper discusses quality driven strategies that will enable you to launch apps quickly, within budget and time.

1. sqs.com
Whitepaper
SQS – the world’s leading specialist in software quality
Effective Test Management can
help you to launch mobile pay-
ments faster, smarter and cheaper
Ensuring Quality Assurance is implemented in complex systems
Introduction
The primary role of test management is to ensure that a quality
assurance and testing programme is conducted efficiently
and effectively. In principle, the role of test management for
m-payment systems is the same as that for traditional systems,
in practice however, there are important differences that can
mean the difference between success and failure.
The many combinations of mobile devices, operating systems
and infrastructure create a logistical nightmare and not all
organisations will have capacity to test all possible permutations.
A 2013 global survey of 1,500 executives1
found that over
56 % of respondents cited the lack of specialised methods
and processes as the biggest barrier to mobile testing in
their organisation. By balancing the variables to be tested
against risk, an m-payment system test strategy can ensure
that sufficient testing will be conducted within reasonable
timescales and budget levels.
Further, a testing programme for mobile succeeds by focussing
on the unique characteristics of mobile devices. Considerations
such as battery consumption, network hand-offs between mobile
and Wi-Fi networks, required levels of security as well as user
interface and App Store delivery need to be considered within
the overall test strategy.
This paper presents a pragmatic, risk-based approach to testing
m-payments systems including issues such as organisation
structure, development methodology and project risk.
Author: Sven Euteneuer
Global Head of Technical Quality
sven.euteneuer@sqs.com
SQS Software Quality Systems AG, Germany
Published: March 2014

2. Page 2© SQS Group 2014
Whitepaper | Effective Test Management can help you to launch mobile payments faster, smarter and cheaper
There are three primary considerations to be addressed in
mobile application testing programmes:
• Complexity: the complexity of the software, hardware and
infrastructure involved
• Quantity: the number and variety of system configurations
to be tested
• Characteristics: the characteristics of the system that need
to be tested (SQS uses the term ‘quality fingerprint’ to
describe this)
1.1. Complexity of m-payment systems
Studies have shown that there is often a strong correlation
between the complexity of software and the number of defects
in that software2
. Despite their apparent simplicity, mobile
applications are often more complex than their desktop or web-
based counterparts while offering fewer features and functions
to end users. To understand the complexity of m-payments
systems it is necessary to look at the system as a whole, not
just the ‘app’. Outside the device, links to back-end systems
and network connectivity add a layer of complexity, while inside
the device, hardware, software and operating system variations
result in mobile systems that are typically much more complex
than traditional software development projects. This complexity
makes effective test management and testing practices in
m-payments crucial.
The issue of complexity and, by implication, quality, applies
throughout the whole life of the system, not just the initial
release. Research by the Rochester Institute of Technology3
,
which used lines of code (LOC) as a measure of complexity,
highlighted how mobile applications are not only more complex
than traditional equivalents, but also become more complex
more rapidly when new versions are released. The research
showed that mobile applications demonstrated a rapidly
accelerating growth in complexity as new versions of the
applications were released, while desktop equivalents tended
to towards a steady growth in complexity.
1.2. Quantity and variety of testing
configurations
Complexity of the payment system itself is not the only issue,
the sheer number and variety of mobile platforms upon which
mobile applications must operate presents a major quality
management challenge.
Test strategies for m-payment systems need to support testing
on multiple mobile devices across multiple platforms, but
they must do so pragmatically. Consider: fully testing 420
device models over 29 Android versions would require 12,180
combinations to be tested for each test case. Even with test
automation, which can dramatically reduce the effort required,
is testing all 12,180 combinations just for Android worthwhile?
Ensuring a secure and reliable customer experience against
the backdrop of a technically complex and diverse mobile
environment is one of the key challenges when planning quality
assurance activity.
1.3. Characteristics of m-payment system
quality – the ‘quality fingerprint’
To many the notion of quality is somewhat ethereal, yet to define
a test concept for m-payments, it needs to be ‘operationalised’,
i.e. there must be a way to measure it and determine when the
desired degree of quality is achieved. Quality models are used
to accomplish this and the ISO/IEC 25010 model is used as
the de-facto standard in the IT industry. This model uses quality
attributes to subdivide the general notion of quality into more
tangible sub-characteristics (Figure 1, next page).
As for every system, the specific combination and prioritisation
of these characteristics varies and can be thought of as a kind
of fingerprint. Just as fingerprints differ between virtually any
two people, so too does, the specific combination of quality
characteristics for an IT system – the “quality fingerprint”.
1. The mobile quality challenge

3. Page 3© SQS Group 2014
Whitepaper | Effective Test Management can help you to launch mobile payments faster, smarter and cheaper
Often the sub-characteristic of functionality alone is targeted
by testing activities, but for m-payments systems, this is simply
not enough. The quality fingerprint for m-payments stresses
the importance of non-functional attributes such as security,
reliability, portability and functional stability. To address the
‘mobile quality fingerprint’, a test strategy should also consider
additional test objects such as networking, user interfaces,
mobility and software delivery mechanisms as highlighted
below.
Figure 2: ISO 25010 Software product quality model
Table 1: Additional test objects for mobile systems
• Network throughput
• Fidelity
• Network hopping,
handover
Network constraints
• Differing screen
sizes and resolutions
• Touch based
• No mouse
• Soft-key only
User interface
• (Closed) app store
• Install via central
download
• Mobile software
product lifecycle
Software delivery
• Finite battery life
• Need for frugal use
of resources such as
CPU
• Interruptions
Mobility
• Appropriate-
ness
• Accuracy
• Compliance
• Availability
• Fault
tolerance
• Recoverability
• Compliance
• Time-
behaviour
• Resource-
utilisation
• Compliance
• Appropriate-
ness
• Recognisability
• Learnability
• Ease of use
• Helpfulness
• Attractiveness
• Technical
accessibility
• Compliance
• Confidentiality
• Integrity
• Non-
repudiation
• Accountability
• Authenticity
• Compliance
• Replaceability
• Co-existence
• Inter-
operability
• Compliance
• Modularity
• Reusability
• Analyzability
• Changeability
• Modification
stability
• Testability
• Compliance
• Portability
• Adaptability
• Installability
• Compliance
Software product quality
Reliability
Perfor-
mance
efficiency
Functional
suitability Operability Security
Com-
patibility
Maintain-
ability
Transfer-
ability

4. Page 4© SQS Group 2014
Whitepaper | Effective Test Management can help you to launch mobile payments faster, smarter and cheaper
Figure 3: Diversity of platforms, versions and devices in m-payments systems
In traditional software development, risk-based planning can be
applied to determine the degree to which functionality should
be tested – which features and functions to prioritise and which
aspects of the system can be simply ‘smoke tested’. However,
when applied to mobile systems, the risk-based approach needs
to be broadened to include platforms, operating systems and
versions for each device the application will run on. Additionally,
it needs to consider all test tasks that arise from the broader
scope of the mobile quality fingerprint. This broader approach
to risk-based testing facilitates the pragmatic allocation of
people and infrastructure, helping to deliver a product that
has been sufficiently quality assured – while at the same time
keeping testing costs from skyrocketing.
Multiple Distribution Channels
Vendor
Dozens of Target Platforms
Hundreds of Devices
SQS’ strategic solution reduces testing effort by 80 %
for a major mobile payments platform
SQS’ proven experience in m-payments gave a leading
financial services organisation the confidence to appoint
them as a strategic partner for its complex multi-device
retail mobile banking services.
Developing a test strategy and environments was chal-
lenging as testing needed to include six major operating
system versions across both iOS and Android devices. SQS
built a test infrastructure to support an offshore testing
model, enabling the team to execute tests remotely without
the need for on-shore resource.
SQS implemented test automation to increase the pace
and quality of testing: the designed environments and
infrastructure enabled the client to execute each test cycle
of 2,000 test scripts in just 30 person-days, a saving of 80 %
compared to equivalent manual testing.
Live Example

5. Page 5© SQS Group 2014
Whitepaper | Effective Test Management can help you to launch mobile payments faster, smarter and cheaper
Risk can help determine where resources are focused, but
risk-based testing of itself is not enough. Speed, agility and
efficiency in test management and development practices
are key to meeting the challenge of delivering systems in the
dynamic and complex mobile marketplace.
Achieving the requisite speed, agility and efficiency, means
underpinning testing practices with a systematic, process-
based testing approach. With such an approach in place, test
cases and other assets are created efficiently and the rate
at which those assets are produced also rises. Re-using test
assets also makes automation simpler – an important benefit
when dealing with regression testing across mobile devices
and platforms.
Adopting a systematic or disciplined approach to test manage-
ment does not remove flexibility from the quality assurance
process. Regardless of whether Agile or Waterfall methods are
employed, test management needs to be tailored so that it
provides the flexibility required to fit in with the development
methodology adopted.
Best practices such as early error detection and the ‘shift left’
of testing (i.e. testing tasks being performed earlier in the
development lifecycle) are typically, though not exclusively,
seen in Agile development projects. Bringing these practices
into an m-payments development project will increase effective-
ness and ultimately reduce cost.
2. Test organisation and development methodology

6. Page 6© SQS Group 2014
Whitepaper | Effective Test Management can help you to launch mobile payments faster, smarter and cheaper
Project and programme owners sometimes regard testing as
a burden that slows delivery. However efficient and effective
testing not only supports product delivery, but also accelerates
the delivery of a quality product. The testing team’s ability to
deliver effective and efficient testing that will increase the speed
of product delivery is largely determined by the tools, processes
and strategy adopted – in short, the test management
approach.
3.1. Doing the right thing: testing
effectively
Test managers need to think differently about mobile testing
compared to traditional software testing projects: methods and
processes need to be modified to make testing more effective
without huge increases in budgets.
But what needs to change in test management for m-payments?
Risk needs to be used much more comprehensively in prioritising
testing effort. By expanding the concept of risk-based testing,
quality assurance effort can be laser-focused to mitigate the
biggest risks and so maximise the testing budget.
Minimising product risk within the available testing budget
means prioritising effectively. How? By extending risk-based
testing to include the unique characteristics of m-payment
systems. Applying risk management at all levels enables test
and development teams to allocate resources effectively. The
chart below illustrates how risk and priority drive the effort
allocated to testing, for example, while critical functional test
cases will be executed on every relevant device at least once,
test cases with a lower than moderate risk and priority might
only be executed once on a single device. Less frequently
used device configurations can be risk assessed by conducting
‘smoke tests’ through crowd-based testing services while higher
priority requirements, such as security, can be addressed by
in-depth security testing.
3. Test effectiveness and efficiency
Figure 4: Prioritising test effort by risk and priority
Priority
1 All
A
All
B
All
C
2 All All Once
3 All Once Once
Risk
Each test case is executed
at least once
on one device.
Each test case is executed
on every relevant device
at least once.

7. Page 7© SQS Group 2014
Whitepaper | Effective Test Management can help you to launch mobile payments faster, smarter and cheaper
3.2. Testing more by testing efficiently
Prioritising testing tasks helps to reduce the overall volume of
testing work. Nevertheless, there is still a clear need to conduct
testing efficiently and make best use of available resource
while still achieving the desired quality assurance outcomes.
There are several strategies to improve testing efficiency - two
of the most effective approaches for m-payments are testing
earlier in the development cycle and using both test tools and
automation frameworks.
3.2.1. Testing earlier: shifting test activities left reduces
cost and effort
Shifting testing left introduces test activities earlier in the
development lifecycle. Opportunity cost, re-test effort and
correction effort are all typically much higher the closer a
project or an agile iteration is to go-live. Identifying and eli-
minating a defect early is almost always cheaper and faster
than addressing that same defect later.
The benefit of shifting left can be realised across all types
of testing, from functional requirements that describe what
the product does to non-functional requirements that define
qualities such as security, performance and scalability.
An example of the value of shifting test activity left can be
found when addressing one of the top considerations in
m-payments for both end-users and providers – security.
Tasks such as threat modelling, static analysis and dynamic
analysis can often be conducted early in the project and help
weed out conceptual as well as technical flaws that would have
turned into real-life vulnerabilities. Carrying out security-testing
tasks earlier in the lifecycle helps to minimise the risk of un-
expected vulnerabilities being discovered late in the project
and incurring the corresponding cost of rework and/or delay.
Conducting testing activity earlier in the lifecycle does not
necessarily mean that testers need to execute test cases
against software delivered by the development team.
Techniques such as document review, developer testing or
using analysis tools are all capable of identifying errors early,
well before the start of classic testing activity.
3.2.2. Tools and automation increase speed and
reliability
A key target for test automation is testing the functionality of
the user interface (the GUI) where the speed and reliability of
automated testing can yield significant benefits, particularly
when testing multiple devices and configurations.
Other tasks that can be automated include:
• Test environment provisioning
• Test data provisioning
• Security testing
• Reliability testing
However, the return on investment when implementing auto-
mation for these tasks should be examined carefully before
proceeding. Test environments that support multiple device
and operating systems can require a significant investment. It is
important to assess the benefits of mobile test tools and mobile
test automation products against the total cost of ownership.

8. Page 8© SQS Group 2014
Whitepaper | Effective Test Management can help you to launch mobile payments faster, smarter and cheaper
SQS’ 2012 iqnite survey found that testing accounts for
a significant portion of project budget4
, while work at the
University of Cambridge found that on average, developers
spend 50 % of their time finding and fixing bugs5
. Ensuring
quality can be a costly undertaking; the key is to conduct
quality assurance and testing effectively and efficiently.
Why do we spend so much time and effort on testing and
quality? Not just for the sake of creating a quality product
per se, but because creating a quality product makes sound
financial sense. For m-payments, where the risk of financial and
reputation loss is much higher than with most other systems,
creating high quality software saves money: it reduces risk
by ensuring that the end product is mature, reliable, cheaper
to maintain and fit for purpose. No organisation wants to be
subjected to the public and customer relations disaster of a
failed m-payments system and effective test management will
help to minimise the risk of such an event ever occurring.
SQS has been providing test management and test execution
services to corporates across the globe for over 30 years.
With a track record of innovation in quality assurance, SQS
consultants are leading the drive towards more efficient and
effective testing of new technologies such as m-payments
through techniques such as better risk management, test
automation, environment management and security testing.
4. Conclusion

9. Page 9© SQS Group 2014
Whitepaper | Effective Test Management can help you to launch mobile payments faster, smarter and cheaper
5. References
1) World Quality Report 1013-14, http://www.capgemini.com/resources/world-quality-report-2013-14
2) Empirical analysis of CK metrics for object-oriented design complexity: implications for software defects,
Subramanyam. R, Krishnan, M.S., 2003
3) The evolution of mobile apps: an exploratory study by Zhang, Sagar & Shihab, 2013.
4) Up to 30 % of project budget in 69 % of new projects: SQS iqnite survey 2012
5) http://www.prweb.com/releases/2013/1/prweb10298185.htm
© SQS Software Quality Systems AG, Cologne 2014. All
rights, in particular the rights to distribution, duplication,
translation, reprint and reproduction by photomechanical or
similar means, by photocopy, microfilm or other electronic
processes, as well as the storage in data processing systems,
even in the form of extracts, are reserved to SQS Software
Quality Systems AG.
Irrespective of the care taken in preparing the text, graphics
and programming sequences, no responsibility is taken for
the correctness of the information in this publication.
All liability of the contributors, the editors, the editorial
office or the publisher for any possible inaccuracies and
their consequences is expressly excluded.
The common names, trade names, goods descriptions etc.
mentioned in this publication may be registered brands or
trademarks, even if this is not specifically stated, and as
such may be subject to statutory provisions.
SQS Software Quality Systems AG
Phone: +49 2203 9154-0 | Fax: +49 2203 9154-55
info@sqs.com | www.sqs.com