Studies show that at least half of all software defects are rooted in poor, ambiguous, or incomplete requirements. For decades, testing has complained about the lack of solid concrete requirements, claiming that this makes our task more difficult and in some instances—impossible. Lloyd Roden challenges these beliefs and explains why having detailed requirements can be at best damaging and at worst can even be harmful to both testing and the business. Rather than constantly complaining, Lloyd shows how testers and test managers can rise to the challenges of testing without requirements, testing with evolving requirements, testing with vague requirements, and testing with wrong requirements. To help make your testing more effective, Lloyd provides practical tips and techniques for each of these testing challenges.

1. T13
Requirements
5/8/2014 1:30:00 PM
Top Challenges in Testing
Requirements
Presented by:
Lloyd Roden
Lloyd Roden Consultancy
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Lloyd Roden
Lloyd Roden Consultancy
With more than twenty-eight years in the software industry, Lloyd Roden has worked as a
developer, test analyst, and test manager for many different organizations. Lloyd was a
consultant/partner with Grove Consultants for twelve years. In 2011 he created Lloyd Roden
Consultancy, an independent UK-based training and consultancy company specializing in
software testing. Lloyd’s passion is to enthuse, excite, and inspire people in the area of software
testing. He has spoken at conferences worldwide including STAREAST, STARWEST, Better
Software, EuroSTAR, AsiaSTAR, and Special Interest Groups in software testing in several
countries. In 2004, he won the European Testing Excellence award.

3. Top challenges in
testing
requirements
Keynote Presentation
Written by
Lloyd Roden
www.lloydrodenconsultancy.com
Version 1_0
© Lloyd Roden

4. © Lloyd RodenTesting Requirements 131014
testing requirements-1
Top Challenges in Testing
Requirements
1
Contents
Introduction
Top challenges for testing requirements
Rising to the challenge
2

5. © Lloyd RodenTesting Requirements 131014
testing requirements-2
What is a requirement?
Definition:
! something that is required/needed
! something essential to the existence
of something else
why do we have requirements?
to make money
requirement for a
new product
to improve
requirement for an
existing system
to remain relevant
requirement for
changing technology
3
What do requirements look like?
specifications
use cases
user stories
story boards
emails
verbal
some forms are better than others
4

6. © Lloyd RodenTesting Requirements 131014
testing requirements-3
Root Cause Analysis shows…
56%
27%
10% 7%
Defects Originate in Project Phase
Requirements
Design
Code
Other
similar statistics from other projects
• 58% requirement bugs for large mobile phone company
• 73% requirement bugs for large financial company
Source: Binder & Associates
5
Problems that we have with requirements
" non-existent requirements
" expectations are not fully understood
" evolutionary requirements
" constantly changing the scope of the project
" wrong or bad requirements
" wasted time in development and testing
" vague or ambiguous requirements
" leading to misunderstandings
" too much detail in the requirements
" lack of creativity
" too complex requirements
" unable to test effectively
" too many requirements
" overbearing and frustrating 6

7. © Lloyd RodenTesting Requirements 131014
testing requirements-4
Contents
Introduction
Top challenges for testing requirements
Rising to the challenge
7
Challenge #1: No requirements
" can we test with no requirements?
" we do it all the time
" performance, usability…
" should we test with no requirements?
" challenge is knowing whether something is right
“I know it when I
see it”
“I want something,
but I don’t know
what”
“and by the way,
could you give me
an estimate”
8

8. © Lloyd RodenTesting Requirements 131014
testing requirements-5
requirement
Challenge #2: Evolutionary requirements
requirement
idea
design
code
test
9
The good and bad of evolutionary requirements
" excellent when
customers don’t really
know what they want
" often produces systems
that meet customer’s
expectations
" high collaboration
" documentation can be
sparse/non-existent
" tester’s nightmare
" automation usually not
possible until final
iteration
10

9. © Lloyd RodenTesting Requirements 131014
testing requirements-6
Challenge #3: Bad or wrong requirements
some bad products…
• all started out as “bad” ideas/requirements
• time “wasted” testing them
we need to recognise and challenge any bad/wrong requirements
11
Challenge #4: Vague/ambiguous requirements
which do you consider to be “good” requirements
" the help messages must be meaningful and informative
" the fields provided must all have context sensitive help
" the system must be user-friendly
" the system must be reliable
" navigation must be consistent across all applications
" the system updates must be fast
" the website must sustain 5000 simultaneous users
" the mobile phone application must be secure
" developers must be able to fix a bug within 2 days
why are some requirements better than others?
12

10. © Lloyd RodenTesting Requirements 131014
testing requirements-7
What is wrong with these requirements?
" requirement 1
" requirement 2
online orders can be made at anytime, but will be processed on the next
working day. Payment must be made using PayPal or standard credit/debit
cards. Orders will be dispatched within 2 days. Customers will be notified via
email if a partial order has been sent and can return their order within 28 days
and obtain a full refund.
working day standard
within
partial within
full
A computer program plays chess with one user at a time. It displays the board
and pieces on the screen. Moves are made by dragging the pieces
13
Challenge #5: Too much detail in the requirement
" is there such a thing of having too much detail
in requirements?
" why are too much detail in requirements a
challenge?
" it often prevents creativity in development and test
" mistakes are made due to a lack of questioning
" too much detail could lead to conflicting
requirements
14

11. © Lloyd RodenTesting Requirements 131014
testing requirements-8
Challenge #6: Complex requirements
and we call this progress…
1990 2013
1990 2013
1990 2013
“yo, I am
outside
your door. lol
15
Challenge complexity at every opportunity
" simplicity seen as weak and uninteresting
" who wants a “basic mobile phone?”
" complex is seen as good
" I don’t understand this,
so it must be really good
(everyone else understands)
" $1m pen
suggestion: challenge requirements and design
documents at every opportunity to see whether
complexity is needed 16

12. © Lloyd RodenTesting Requirements 131014
testing requirements-9
Features and functions used
Jim Johnson XP2002 Standish Study Group
Features and Functions Used
16%
13%
7%
19%
45%
Sometimes Used
Often Used
Always Used
RarelyUsed
Never Used
Features and Functions Used
20%
64%
16%
Often and Alw ays
Used
Rarely or Never Used
Sometimes
this means we have driven up
complexity by putting in things
that are not required
17
.. If there is too little demand on them, people are bored.
If there is too much for them to handle, they get anxious. Flow
occurs in that delicate zone between boredom and anxiety.
Performance
Arousal level
Flow
Strained
concentration
Boredom
Distracted
Csikszentmihalyi
Challenge #7: Too many requirements
low medium high
lowhighmedium
Stress
Laid back
Anxious
Panic/Anger/Fear
Optimum
Performance
18

13. © Lloyd RodenTesting Requirements 131014
testing requirements-10
MoSCoW
" Must
" describes a requirement that must be satisfied in the final
solution for the solution to be considered a success
" Should
" represents a high-priority requirement that should be in the
solution if possible. It is often a critical requirement but one
which can be satisfied in other ways if necessary
" Could
" describes a requirement which is considered desirable but
not necessary. This will be included if time and resources
permit
" Wont
" represents a requirement that stakeholders have agreed will
not be implemented this time but may be considered for the
future 19
Contents
Introduction
Top challenges for testing requirements
Rising to the challenge
20

14. © Lloyd RodenTesting Requirements 131014
testing requirements-11
Choose the most appropriate lifecycle
Integration
Testing
Acceptance
Testing
System
Testing
Component
Testing
Code
Design
Specification
System
Specification
Business
Requirements
sequential iterative agile
• detailed requirements
• complex requirements
• vague requirements
• no requirements
• evolutionary requirements
• vague requirements
• too many requirements
• complex requirements
• vague requirements
there is no lifecycle applicable for bad/wrong requirements
21
" an agile development approach to component
testing
Use Test-Driven Development
pick a
req.
yesno
need more tests
for requirement? write enough code
to pass the test
write component
test
automate test
using a tool
22

15. © Lloyd RodenTesting Requirements 131014
testing requirements-12
Use test design techniques as a review technique
" example requirement:
by reading this requirement we may have a few
questions, or we might think we understand what is
required. However…
A thermostat displays the temperature of a greenhouse from
15o
C to 35o
C. If the temperature falls below 20o
C then “too cold”
appears on the display and a blue light flashes. It the
temperature exceeds 30o
C then “too hot” appears and a red light
flashes.
23
Review using test case design techniques
" sharpens our understanding
" generates tests
" finds more defects with requirements
Tests for our example:
Temperature Expected result
17oC, 15oC, Too Cold, blue light flash
20oC, 25oC, 30oC No display, no light?
32oC, 35oC Too hot: red light flash
10oC, 40oC ???
2 lights or 1 light?
whole degrees?
up to and including…
where is the temperature taken…isolated or whole room?
24

16. © Lloyd RodenTesting Requirements 131014
testing requirements-13
Produce some requirement guidelines
do not use pronouns
do not use unqualified adjectives and adverbs
do not use the word “etc”
is the requirement is testable?
do we have the necessary expertise?
are the requirements prioritised?
does the requirement need breaking down?
is the requirement clear and understood?
25
Use walkthroughs to clarify and understand
" walkthroughs are a powerful static testing technique
" to educate, aid understanding and find bugs in documents
" the power and full potential of the walkthrough is
often not recognized or utilized
" watch for what is said and how it is said
" do not underestimate body language
" resist peer pressure and challenge when
you don’t understand
26

17. © Lloyd RodenTesting Requirements 131014
testing requirements-14
Use the Quality Attribute Template
Concept Test Scale Plan Must Now Best
completeness compare
menu and
help items
percentage
of menu
items in the
help
100% 90% 40% 100%
ease of
access
sample 25
features
average
number of
keystrokes
to find a
feature
3 7 12 2
accuracy collect
data each
month
number of
reported
defects in
the help
<5 20 50? 0
Usability
On-linehelp
high
level
attribute
sub
attribute
description
of attribute how
is
it to
be
m
easured what
is
being
m
easured
m
easure
to
be
achieved worse
m
easure
acceptable
current
m
easure
ultim
ate
m
easure
particularly useful for vague non-functional requirements 27
Perform Exploratory Testing
" it is not obvious what the next test
should be
" we want to go beyond the obvious
tests
" we want to assess the product
quickly
" we want to explore areas of the
system that are unclear
" creative skills in testing are being
encouraged
" we don’t know all the detail of the
requirements
ET is useful when…
28

18. © Lloyd RodenTesting Requirements 131014
testing requirements-15
Summary
" various forms of
requirements exist
" there are many challenges
that we are often faced with
when testing requirements
" testing must rise to these
challenges by using a
variety of techniques,
methods and tools in order
to be as effective as we can
in testing the requirements
29