GLA Testing Presentation by Test Partners Ltd v1

Software Testing
Presentation
October 2017

Introduction
• Founded in 2001
• Test Lab in Central London
• Specialise in testing web-based applications and mobile applications
• Functionality, compatibility and accessibility
• 2000+ projects
• 500+ clients
Software Testing Presentation - October 2017Page 2

Software Testing Presentation - October 2017

Today’s Topic
Point 8 of the local government digital service standard:
“Be able to test the end-to-end service in an environment similar to that of the
live version, including all common browsers and devices.”
http://localgovdigital.info/localgov-digital-makers/outputs/local-government-
digital-service-standard/standard/8/

Today’s Topic
I will talk about:
• Comparison of the benefits and shortcomings of manual testing and
automated checking.
• Context-driven testing.
• Scripted and exploratory testing.
• Client-side test environments.
• Online services such as BrowserStack and Perfecto.
• Choosing the devices and browsers to test.
• Approaches to manual regression testing.
• Test data.

Today’s Topic
I will not talk about:
• Unit testing.
• Automated checking (with one small exception).
• Security and load testing.
• Infrastructure and test environments.
• Development approaches such as TDD and BDD.
• Accessibility testing (but perhaps another time).

Key Requirements
From the standard:
• Test not only during development but throughout the life of the service.
• As services will be part of a continual improvement review, it is important to
be able to test changes.
• It is important that the service is tested on the devices and internet browsers
that we know are in use within the user base.

Comparison of the benefits and shortcomings of manual testing and automated
checking
Criterion Manual Testing Automated Checking
Objective Investigate the
application to verify
expected behaviours,
identify unexpected
behaviours and
identify any other
information of interest
to stakeholders.
Verify expected
behaviours.

checking
Speed (initial test) Slow, but produces
results immediately
from the start.
Slower than manual
testing due to scripting
time. Typically no
results for some time.

checking
results immediately
from the start.
Slower than manual
time. Typically no
Speed (subsequent
tests)
Much the same as the
initial test.
Much faster than the
initial test.

checking
results immediately
from the start.
Slower than manual
time. Typically no
Speed (subsequent
tests)
Much the same as the
initial test.
Much faster than the
initial test.
Ability to
accommodate change
Exploratory testing
easily accommodates
change. Scripted tests
need to be updated.
Poor - requires script
maintenance.

checking
Thoroughness (test
scenarios)
Typically includes both
simple and complex
scenarios.
Typically limited to
simple "happy path"
scenarios.

checking
Thoroughness (test
scenarios)
simple and complex
scenarios.
simple "happy path"
scenarios.
Thoroughness (test
data)
Very limited. Can check a huge
number of data
variations.

checking
Thoroughness (test
scenarios)
simple and complex
scenarios.
simple "happy path"
scenarios.
Thoroughness (test
data)
Very limited. Can check a huge
number of data
variations.
Performs "what if…"
tests
Exploratory testing
does but scripted tests
typically do not.
No. Automated tools
cannot ask questions
or apply intelligence.

checking
Notices behaviour that
was not the focus of
the test.
It is an important
feature of exploratory
testing. People doing
scripted tests could
but typically do not.
No. Automated checks
only verify the
assertions they have
been programmed to
do.

checking
Notices behaviour that
was not the focus of
the test.
It is an important
feature of exploratory
testing. People doing
scripted tests could
but typically do not.
No. Automated checks
only verify the
assertions they have
been programmed to
do.
Ability to test without
design documentation
No problem at all for
exploratory testing.
Scripted tests require
design documentation.
Automated checks
require design
documentation.

Context-driven Testing
Your context includes factors such as:
• Objectives
• Timescale
• Available budget and other resources
• Risk
• Scope and nature of changes
• Regulatory environment
• Lifespan
• Constraints
• Organisational culture

Context-driven Testing
• Start by identifying your context.
• Consider how your context should influence your approach to testing.
• Be aware when your context changes. Do you need to change your testing?
• Be wary of “best practices” that are implemented without consideration –
these cause:
• Waste.
• Risk.
Sounds obvious but it rarely happens in practice. People tend to do what they
have always done.
http://context-driven-testing.com/

Different Types of Testing
• Give you different things.
• Automated checking is not better than manual testing. It’s totally different.
• Exploratory testing is not necessarily better than scripted testing (although it
is for most purposes).
• Choose the types of testing that are most appropriate in your context.

Scripted and Exploratory Testing
• Scripted testing focuses on verifying the documented requirements.
• Cannot test without documented requirements.
• Vulnerable to intentional undocumented design changes.
• Exploratory testing focuses on what users can do to the system.
• Verifies expected behaviours.
• Identifies unexpected behaviours.
• Little or no design documentation required.
• Can accommodate incomplete functionality and undocumented design
changes.

Criterion Scripted Testing Exploratory Testing
Skill level required Minimal. Anyone can do it but
requires skill to do
well.

well.
Speed Very slow. 2 to 3 times faster.

well.
Good for finding bugs Not if your developers
are any good.
Finds heaps of bugs.

well.
are any good.
Records test coverage Yes. Can do.

well.
are any good.
Provides detailed audit
trail
Yes. Can do but adds
overhead.

well.
are any good.
Provides audit trail Yes. Can do but adds
overhead.
Fun level Tedious beyond words. Highly engaging.

Client-side Test Environments
If you want accurate, repeatable results:
• Test on dedicated “clean” test machines i.e. new installation of the operating
system and nothing else.
• Avoid emulators.
• Avoid virtual machines such as Parallels, VMWare and VirtualBox.
• Don’t use tools that run multiple browser versions in one browser.
• Check the provenance of all sources of browsers and plug-ins. Build a library
while new versions are still available.
If you value speed, cost and convenience over accuracy and repeatability, feel
free to ignore all the above.

Client-side Test Environments
SaaS services include:
• BrowserStack - https://www.browserstack.com/
• Perfecto Mobile - https://www.perfectomobile.com/
Advantages
• Cheap compared with buying devices.
• Easier than maintaining a software library.
• Probably ok for automated checking.
Disadvantages
• My personal experience was that they were terrible for manual testing.

Choosing Which Browsers and Devices to Test
The standard says “analyse current user behaviour to determine the device and
browser types in use”.
• OK up to a point. But if the website does not work well with that browser or
device, its browser or device usage statistics will be disproportionally low.
• Use other sources of statistics but:
• Check the demographics. Do they match yours?
• Check the methodology. Is it valid?
• Check the time period the statistics were collected over.
• A narrow time period may not be statistically valid.
• A wide time period will give emphasis to older browsers.
• I usually advise one week to one month.

Choosing Which Browsers and Devices to Test
There are plenty of free sources but the best data usually has to be paid for:
• http://marketshare.hitslink.com/
• https://mixpanel.com/trends/#report/ios_frag/from_date:0,to_date:0
• http://www.w3schools.com/browsers/browsers_stats.asp (techie-biased)
• https://www.fiksu.com/resources/ios_trackers
• http://opensignal.com/reports/2015/08/android-fragmentation/
• https://www.browserstack.com/test-on-the-right-mobile-devices
• http://gs.statcounter.com/
• https://www.appbrain.com/stats
• https://david-smith.org/iosversionstats/

Manual Regression Testing
There is no “right” answer. Possible strategies include:
• Always do the same tests with the same test data.
• Always do the same tests with different test data.
• Do different tests each time.
• A mixture of these.
• You stand the best chance of finding a bug if you do something you have not
done before.
• You might have a core set of tests, but introduce as much variation as you
can.

Manual Regression Testing
More options:
• Execute all the same “small” tests that you initially did.
• Very time-consuming.
• Potentially wasteful.
• Execute a subset of the same “small” tests that you initially did.
• Saves time but is potentially risky.
• Write a few “mega” tests that touch all the important features and areas.
• Efficient. Frees up time to do more targeted testing.
• One bug can block all subsequent tests.
• Complex scenarios can make fault diagnosis difficult.

Test Data
When entering data in forms:
• Don’t just “bang on the keyboard”.
• Create data that helps you identify if a bug has occurred.
• Enter unique data wherever possible.
• Make the data self-describing.
• Keep a record of the data you enter, with time and date stamps.
Benefits
• Wherever or whenever you see a piece of data, you know where and when
you entered it.
• Corrupted and misplaced data is easily identified.

Test Data
Example of typical data used when registering accounts for 10 users of an
e-commerce system.
Use Excel’s concatenation function to automatically create more complex data.
Automatic data creation reduces probability of errors.
Date Time Title First Name Last Name Email Address 1 Address 2 Address 3 Town County
12 June 2017 09:37 Mr first1 last1 1sdg@testpartners.co.uk Address1-1 Address2-1 Address3-1 Town1 County1
12 June 2017 10:12 Mrs first2 last2 2sdg@testpartners.co.uk Address1-2 Address2-2 Address3-2 Town2 County2
12 June 2017 10:18 Miss first3 last3 3sdg@testpartners.co.uk Address1-3 Address2-3 Address3-3 Town3 County3
13 June 2017 14:34 Ms first4 last4 4sdg@testpartners.co.uk Address1-4 Address2-4 Address3-4 Town4 County4
13 June 2017 16:05 Dr first5 last5 5sdg@testpartners.co.uk Address1-5 Address2-5 Address3-5 Town5 County5
13 June 2017 16:17 Mr first6 last6 6sdg@testpartners.co.uk Address1-6 Address2-6 Address3-6 Town6 County6
13 June 2017 17:01 Mrs first7 last7 7sdg@testpartners.co.uk Address1-7 Address2-7 Address3-7 Town7 County7
14 June 2017 10:33 Miss first8 last8 8sdg@testpartners.co.uk Address1-8 Address2-8 Address3-8 Town8 County8
14 June 2017 12:55 Ms first9 last9 9sdg@testpartners.co.uk Address1-9 Address2-9 Address3-9 Town9 County9
14 June 2017 13:42 Dr first10 last10 10sdg@testpartners.co.uk Address1-10 Address2-10 Address3-10 Town10 County10

Steve Green
Managing Director
Test Partners Ltd
68-72 Queen Victoria Street, London EC4N 4SJ
020 3002 4176 (direct)
0800 612 2780 (switchboard)
07957 246 276 (mobile)
020 7692 5517 (fax)
Skype: testpartners
Twitter: testpartners
http://uk.linkedin.com/in/stevegreen2
steve.green@testpartners.co.uk
www.testpartners.co.uk
Page 35
Contact
Software Testing Presentation - October 2017

GLA Testing Presentation by Test Partners Ltd v1

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to GLA Testing Presentation by Test Partners Ltd v1

Similar to GLA Testing Presentation by Test Partners Ltd v1 (20)

More from Unboxed

More from Unboxed (18)

Recently uploaded

Recently uploaded (20)

GLA Testing Presentation by Test Partners Ltd v1

Editor's Notes