This presentation is aimed to stimulate improvement at requirements review, with the intent of improving defect injection. Some specific mention is made of non-functional requirements - specifically performance and security. This is one slide pack of a set.

1. 1
Ian McDonald
Requirements Verification
© 2010, 2012. 2013 Ian McDonald

2. 2
Overview
 This slide set deals with requirements and
early high level assessment. Special mention
is made of non-functional requirements such
as performance and security.
 The presentation is specifically targeted at
functional testers and requirements authors to
stimulate the initial thought process in
considering non-functional requirements, when
no or limited performance or security would
otherwise be considered.

3. 3
A Successful Requirement
This section deals with what is a
successful requirement?

4. 4
Good Requirements
Good Requirements provide:
 Clarity of purpose.
 Limits and boundaries that are defined.
 Clarity and traceability, so:
 Development is quick.
 Testing is fast and effective, mitigating risk.
 Product is fast to market – making the right impact.
 Financial Return is good.
 Maintenance is low.

5. 5
Requirements Test Checks
Get a test lead to check the
requirements over. Typical problems
that can be sorted are:
Structure related – risk of missing important
attributes.
Gaps in requirements – not all options
considered.
Poor attention to performance requirements.
Lack of attention to security requirements.

6. 6
Requirements Fault Injection
 Testing starts with the
requirements.
 A significant cost saving can
be created by ensuring that
the requirements are tested
prior to starting coding.
 The test team will become
the guardians of the
requirements, ensuring that
was is required is delivered.
It is therefore vital that the
test team have a full
understanding of the
requirements and that the
requirements are structured
to assist the test team in
ensuring that the business
vision is delivered.
Typical Fault Origin and detection rates within the industry.
Note: Fault cost assumes an hourly cost of £15.

7. 7
Good Requirements – POWER CO
 Good requirements lead to good products.
Good requirements are:
 Possible - Feasible
 Ordered – Prioritised
 Well Written – Correct
 Essential – Necessary
 Recognisable – Reasoned
 Checkable – Verifiable
 On its Own – Single Logical Statements

8. 8
Possible - Feasible
 Write requirements that are possible to implement
within the capabilities and limitations of the system
and environment.
 Avoid unrealistic requirements such as “instantly” or
“immediately”.
 Avoid infeasible requirements, by having a developer
work with the requirements analysts or marketing
personnel throughout the requirement creation
process. The developer will provide a reality check on:
 What can and cannot be done technically,
 What can only be done at excessive cost or with other
tradeoffs.

9. 9
Ordered – Prioritised
 Assign a priority to each requirement, feature, or use case to indicate how essential it is to
include it in a particular product release.
 If delivering within an Agile lifecycle then delivery time is fixed and product content sacrifices
are made.
 Customers or their representatives have responsibility in establishing priorities.
 When the requirements are regarded as equally important, the project manager is less able
to react to:
 New requirements added during development,
 budget cuts,
 schedule overruns
 departure of a team member.
 Priority represents a measure of the value provided to the customer, relative cost of
implementation and the technical risk of implementation.
 Priority levels are typically:
 Critical priority - requirement must be incorporated in the next product release even if
this means a delay. Ideally this will be no more than 10% of the total number.
 High priority - requirement to be incorporated in the next product release, but should not
cause a delay to go live. Ideally this will be no more than 20% of the total number.
 Medium priority - requirement is necessary but it can be deferred to a later release.
Typically this will be no more than 40% of the total number.
 Low priority – requirement is a nice to have, but it can be dropped if insufficient time or
resources.

10. 10
Well Written – Correct
 Describe each individual requirement precisely and
correctly.
 Document the source of the requirement. e.g.
customer reference, higher-level system requirements
specification, change request, etc.
 If a software requirement conflicts with a
corresponding system requirement then an error is
present.
 Only user representatives can determine if a
requirement is correct so for a requirements inspection
include them, or their representatives.
 Otherwise: lead to guessing or incorrect assumptions by
developers and testers:
 This leads to code defects and even a product recall.

11. 11
Essential – Necessary
 Each requirement documents something the
customers needs or something that is required
for conformance to an external requirement,
an external interface, or a standard.
 Each requirement originates from an
authoritative source.
 Each requirement is traced back to its origin.
e.g. use case, system requirement,
regulation, other customer input.
 If no origin, then is the requirement “over
engineering” and not essential?

12. 12
Recognisable – Reasoned
 A good requirements has only one consistent interpretation.
 Multiple readers will have the same interpretation.
 Avoid subjective words like: user-friendly, easy, simple, rapid,
fast, efficient, several, state-of-the-art, improved, maximise, and
minimise. Think what is the logical test that provides evidence of
pass or fail.
 Words that are clear to the requirements author may not be clear
to readers. Write each requirement in clear, simple language,
avoid techno-gabble – if this is necessary, then define it in a
glossary.
 To reveal ambiguity:
 Carry out formal inspections of the requirements specifications,
 Write test cases from requirements,
 Create user scenarios illustrating any expected product behaviour.

13. 13
Checkable – Verifiable
 Each requirement is properly implemented in the
product. This is achieved by:
 Tests which are repeatable with clear pass / fail criteria.
 Verification approaches, such as inspection or demonstration.
 If a requirement is not verifiable, then just a matter of opinion.
The Requirement was of poor quality not the product.
 Requirements must be consistent, feasible,
unambiguous and verifiable.
 Any requirement that says the product shall "support"
something is not verifiable. (Support to what level? Be
specific).

14. 14
On its Own – Single Logical Statements
 A single Requirement will have a single test or group
of tests associated to prove delivery.
 A Change in requirement means a change in test.
 Within the configuration control – build in triggers, otherwise
testing may not test the latest requirements.
 DO NOT write long winded paragraphs sighting great
detail of functionality in one single Requirement
clause. Avoid “AND”, “OR”, “Either”, etc. Single
requirement – single proof (although perhaps many
tests for a single proof set – e.g. boundary values).
 Where necessary break top level requirements (A-
Level) into lower (B-Level) requirements to provide
clarity. Testing all B-levels will automatically mean
that the A-level requirement needs no further testing in
isolation (since all components were tested).

15. 15
Managing Success
This section deals with how to manage
requirements for a successful delivery.

16. 16
Review for Correctness
Is the Requirement Correct?
Is the requirement within the release scope?
Is the requirement logically derived and not
just an arbitrary value?
Is the requirement technically possible?
Is the requirement over-stringent?
Is it a requirement (what) as opposed to a
description of implementation (how)?

17. 17
Review for Ambiguity
Is the Requirement Unambiguous and
Measurable?
A statement that can only be interpreted
one way.
Does not contain weak phrases or a poor
sentence structure that can be
misinterpreted.

18. 18
Review for Atomic, Traceable & Manageable
 Is the Requirement Atomic, Traceable & Manageable?
 Requirement is a single logical statement that cannot be
separated out into ‘smaller’ sub-requirements.
 Contains no logical AND/OR statements.
 All requirements have a unique single identifier that can be
used to map tests and software features to the requirements
from which they were derived.
 All lower sub-requirements are traceable to parent-
requirements.
 An infrastructure is in place to support the requirement and
test traceability.
 Related requirements are grouped together and refer to each
other.
 The requirement structure and style can be preserved even if
the parameter value of the requirement change.

19. 19
Review for Testable & Verifiable
 Is the Requirement Testable and Verifiable?
 Can the requirement be tested with a pass/fail or
quantitative assessment criteria derived from the
specification itself and/or referenced information?
 No subjective statements. Requiring that a system
must be easy to use is subjective and therefore is
not testable.
 Are the valid / Invalid ranges defined?
 Performance requirements state under what
conditions the requirement is true. E.g. is a login
response time to be measured for a single login, or
while 200 virtual users are logging on spread over a
3 minute period?

20. 20
Review for Consistency
Is the requirement set consistent?
A consistent specification has no conflict
between individual requirement statements.
The defined behaviour of essential
capabilities, specified behavioural properties
and constraints do not have an adverse
impact on that behaviour.
 For example: “The only aircraft that is totally safe
is one that cannot be started, contains no fuel or
other liquids, and is securely tied down” (NASA).

21. 21
Review for Prioritisation
Are Requirements Prioritised?
Requirements need to be ranked according
to importance for delivery, based upon the
importance of the overall Business Goal.

22. 22
Review for Impact Assessment
Are Requirements Impact Assessed?
Is the requirement assessed for its impact
[High(3)/Medium(2)/Low(1)] upon the
business if the requirement fails?
 NOTE:
Test Risk Score = (Impact Score) x (Development Assessed Likelihood of Failure Score).
The Development Assessed Likelihood of Failure Score can be assessed by the development
team by assigning a numerical value: e.g.: Extremely Likely on a weekly bases (4), Likely on
a quarterly bases (3), Possible (2), Outside small chance or lower (1).

23. 23
Managing review Feedback
 To optimise the review feedback, consider
maintaining a Review card to mark the
requirement against each criteria (pass/fail).
 The review should be carried out by users, the
test manager and the development manager –
so there is a full understanding and lack of
future requirement blame. If a requirement
later causes a problem then it is the fault of the
review process.
 Identify the mitigation actions required for all
high risk requirements.

24. 24
Non Functional Requirements
This section specifically looks at
performance and security testing

25. 25
Special Case of Non Functional Requirements
 13 non-functional requirements are specifically identified within IEEE-Std 830.
 Performance
 Security
 Interface
 Operational
 Resource
 Verification
 Acceptance
 Documentation
 Portability
 Quality
 Reliability
 Maintainability
 Safety
 Other groupings often identified include:
 Implementation
 Interface
 Operations
 Legal
 Within this presentation 2 specifically are identified for special mention:
 Performance
 Security.

26. 26
Performance Testing Introduction
This section starts to deal with
performance testing. It is intended only
to stimulate thought – where perhaps no
previous consideration has been given
to early performance testing. It does not
cover the details of how to define and
set up tests or in how to interpret results.
It is purely focused on early mitigation.

27. 27
Performance Requirement Types
Performance Criteria often include:
 Availability - The proportion of time a system is in a functioning condition.
 Response time - The interval from when an operator at a terminal enters a request, to the instant at which
the first character of the response is received at a terminal. In data systems, the system response time is the
interval from the receipt of an end of transmission for an inquiry message, to the start of a response
transmission.
 Channel (or Shannon) Capacity - The tightest upper bound on the rate of information that can be reliably
transmitted over a communications channel.
 Latency - The time delay experienced in a system
 Completion time - The amount of time required for a specific task to be completed.
 Service time - The time duration to service a specific item.
 Bandwidth - The measurement of the bit-rate for available or consumed data communication, expressed in
multiples of bits per second (bit/s, kbit/s, Mbit/s, Gbit/s, etc.) for computer networks.
 Capacity - The total measure that can be contained or produced for the system. This can include for
Example Requirement: data traffic or data storage.
 Throughput - The average rate of successful message delivery over a communication channel.
 Efficiency - The resource consumption for a given load.
 Scalability - The ability for a system, network, or process to handle an increased amount of work in an
appropriate manner, or the ability to be enlarged to accommodate growth or demand.
 CPU benchmarks - The performance characteristics for a specific central processor unit. However this can
apply to the running of software on a specific CPU type.
 Resource constraints include processor speed, memory, disk space, network bandwidth, etc.

28. 28
General Principles for Performance Requirements
1. Avoid phrases such as “immediately” or “instantaneous” All
transactions take some time.
2. Avoid requirements that would need control over external
systems. For example to carry out a timing test over the
internet would mean that the traffic would need to be controlled
and kept consistent each time. So consider where the
measurement is to be taken and under what conditions. A
simulated load might be required and the user terminal might
need to be connected at the data centre, for the timing to be
valid.
3. Think about tolerances of readings. E.g. If a timing is to be
measured as 3 seconds, would the test fail if the reading was
either 2.99s or 3.2 sec? Specify either the top or bottom limit
as appropriate. If necessary specify the range or tolerance
permitted.

29. 29
Example Performance Requirements
 The system shall be available 24/7 for 363 days in any one calendar year.
 When an operator requests data, it shall start to be presented on the screen within no more than 3 seconds.
Based upon a normal background scenario running (As described in document XYZ) and the user terminal
being connected to the server.
 The channel capacity for 6.2GHz at a distance of 11m shall be greater than 10-6.
 for a web page download containing 512Kb of text, the latency shall not be greater than 900 ms.
 Upon submitting a request to create a new user profile, the profile shall be created within 2.1 seconds (+/-
0.2s), based upon a normal background scenario running (As described in document XYZ).
 The timing from the moment the credit transaction is submitted to the server, a user information message for
the outcome of the transaction is to be sent to the user within no more than 10 seconds.
 The interface shall transmit data at a minimum rate of 9,600 bits/second.
 The spectral efficiency (or modulation efficiency) is to be not less than:
 18.1 (bit/s)/Hz downstream.
 15.5 (bit/s)/Hz upstream.
 The telecommunications bandwidth shall be not less than 4.5 KHz.
 Application X shall be able to be installed on a Windows 7 Pro system with a minimum of 2Gb of free hard
drive space.
 For an Ethernet connection of between 98 and 101 Mbit/s, the throughput shall not be less than 70 Mbit/s.
 When dealing with current process demand levels (detailed within document ABC), the replacement system
shall be at least 40% more energy efficient than the existing systems (whose energy consumption figures are
contained within document ABC).
 The system shall be scalable to accommodate growth from 1,000 records to 500 million records.
 Game X shall run at 60 +/- 2 frames per second on a system containing a Corei7-2620M 2.7GHz processor
with a HP Elitebook 8510w Nvidia 256mb laptop video graphic card.
 Application X shall function with a minimum of 4GB of RAM on a laptop containing a Corei7-2620M 2.7GHz
processor.

30. 30
General rule for testers
 Do not leave performance till post integration. At
that point it can be too late to fix problems and so
only patching is possible, potentially opening up
security weaknesses. Instead:
1. Create a realistic understanding of the operational
conditions.
2. Design in performance – check the requirements and
design.
3. Review code for performance issues.
4. Test unit code for performance.
5. Test the application for performance.
6. Test the system for performance.

31. 31
Security Section
This section starts to deal with security
testing. It is intended only to stimulate
thought – where perhaps no previous
consideration has been given to early
security testing. It does not cover the
details of security testing or penetration
testing.

32. 32
Security Requirements
 Security requirements seek to ensure:
 Denial of unauthorised access to the system and
the system data.
 Defend the integrity of the system from malicious
(including harmful accidental) damage
 There are at least two basic metric groupings:
 The ability to resist unauthorised attempts at usage
 Continued service to genuine users while the
system is under attack - avoiding Denial of Service
(DoS).

33. 33
Negative and Non-Functional
 Software is often not tested for security. At best the system may
be subjected to a penetration test, which can often be a high level
assessment. To improve security, software must be secure by
design.
 However usually requirements for software security are not
specified. Further, negative and non-functional requirements
may not be documented because they are not considered by
requirements authors as “testable” or “actionable”.
 To design in security from the start, the requirements author
needs to go through a the process of listing the negative and non-
functional requirements that are crucial to security.
 Next the requirements author needs to analyse the negative and
non-functional requirements to determine the resulting functional
requirements required to support security.
 The catch is the requirements authors may not have the skill set
to do this.

34. 34
Early Security Consideration
 When addressing security early consider:
 Assume the system will be hacked. What is the inner
defence?
 Normal entry point may be secure by password, but what if it
is disclosed, or data is stolen by an employee?
 Do all users have access to all data?
 Are there any back door connections to 3rd
party systems that
may be less secure?
 For data entry are the fields open to any data – allowing SQL
injection? Limit data content and size.
 Is there any security monitoring in place – do not rely upon
this – still test applications!

35. 35
Security Risk
 In assessing security risk, the use of metrics is
important, since that will direct the tester to the
greatest need for mitigation.
 Authentication metrics.
 User types and access to data.
 Resistance to known attack types.
 Probability of finding a key – time, effort and
resources needed.
 Percentage of services available during an attack.
 Lifespan of a password.
 Lifespan of a session.

36. 36
Agile response to Non-functional testing
 For the tester, the key to success is to ensure
that non-functional testing starts early.
 Within an Agile delivery approach, this means
including non-functional requirement tests in
the early burn-down lists, ensuring that test
activities are included. The scope of the tests
may well grow at each sprint, but they need to
be included from the start.

37. 37
Mitigating Security Risk
Understand the threats and assume the
worse will happen. What is the next line
of defence.
Review the requirements and
architecture for security.
Review code early for security issues.
Test the application for security issues.
Test the system for security issues.

38. 38
END
Slides created by:
Ian R. McDonald
uk.linkedin.com/in/islandsystems