Requirements Verification v3


Published on

This presentation is aimed to stimulate improvement at requirements review, with the intent of improving defect injection. Some specific mention is made of non-functional requirements - specifically performance and security. This is one slide pack of a set.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Requirements Verification v3

  1. 1. 1 Ian McDonald Requirements Verification © 2010, 2012. 2013 Ian McDonald
  2. 2. 2 Overview  This slide set deals with requirements and early high level assessment. Special mention is made of non-functional requirements such as performance and security.  The presentation is specifically targeted at functional testers and requirements authors to stimulate the initial thought process in considering non-functional requirements, when no or limited performance or security would otherwise be considered.
  3. 3. 3 A Successful Requirement This section deals with what is a successful requirement?
  4. 4. 4 Good Requirements Good Requirements provide:  Clarity of purpose.  Limits and boundaries that are defined.  Clarity and traceability, so:  Development is quick.  Testing is fast and effective, mitigating risk.  Product is fast to market – making the right impact.  Financial Return is good.  Maintenance is low.
  5. 5. 5 Requirements Test Checks Get a test lead to check the requirements over. Typical problems that can be sorted are: Structure related – risk of missing important attributes. Gaps in requirements – not all options considered. Poor attention to performance requirements. Lack of attention to security requirements.
  6. 6. 6 Requirements Fault Injection  Testing starts with the requirements.  A significant cost saving can be created by ensuring that the requirements are tested prior to starting coding.  The test team will become the guardians of the requirements, ensuring that was is required is delivered. It is therefore vital that the test team have a full understanding of the requirements and that the requirements are structured to assist the test team in ensuring that the business vision is delivered. Typical Fault Origin and detection rates within the industry. Note: Fault cost assumes an hourly cost of £15.
  7. 7. 7 Good Requirements – POWER CO  Good requirements lead to good products. Good requirements are:  Possible - Feasible  Ordered – Prioritised  Well Written – Correct  Essential – Necessary  Recognisable – Reasoned  Checkable – Verifiable  On its Own – Single Logical Statements
  8. 8. 8 Possible - Feasible  Write requirements that are possible to implement within the capabilities and limitations of the system and environment.  Avoid unrealistic requirements such as “instantly” or “immediately”.  Avoid infeasible requirements, by having a developer work with the requirements analysts or marketing personnel throughout the requirement creation process. The developer will provide a reality check on:  What can and cannot be done technically,  What can only be done at excessive cost or with other tradeoffs.
  9. 9. 9 Ordered – Prioritised  Assign a priority to each requirement, feature, or use case to indicate how essential it is to include it in a particular product release.  If delivering within an Agile lifecycle then delivery time is fixed and product content sacrifices are made.  Customers or their representatives have responsibility in establishing priorities.  When the requirements are regarded as equally important, the project manager is less able to react to:  New requirements added during development,  budget cuts,  schedule overruns  departure of a team member.  Priority represents a measure of the value provided to the customer, relative cost of implementation and the technical risk of implementation.  Priority levels are typically:  Critical priority - requirement must be incorporated in the next product release even if this means a delay. Ideally this will be no more than 10% of the total number.  High priority - requirement to be incorporated in the next product release, but should not cause a delay to go live. Ideally this will be no more than 20% of the total number.  Medium priority - requirement is necessary but it can be deferred to a later release. Typically this will be no more than 40% of the total number.  Low priority – requirement is a nice to have, but it can be dropped if insufficient time or resources.
  10. 10. 10 Well Written – Correct  Describe each individual requirement precisely and correctly.  Document the source of the requirement. e.g. customer reference, higher-level system requirements specification, change request, etc.  If a software requirement conflicts with a corresponding system requirement then an error is present.  Only user representatives can determine if a requirement is correct so for a requirements inspection include them, or their representatives.  Otherwise: lead to guessing or incorrect assumptions by developers and testers:  This leads to code defects and even a product recall.
  11. 11. 11 Essential – Necessary  Each requirement documents something the customers needs or something that is required for conformance to an external requirement, an external interface, or a standard.  Each requirement originates from an authoritative source.  Each requirement is traced back to its origin. e.g. use case, system requirement, regulation, other customer input.  If no origin, then is the requirement “over engineering” and not essential?
  12. 12. 12 Recognisable – Reasoned  A good requirements has only one consistent interpretation.  Multiple readers will have the same interpretation.  Avoid subjective words like: user-friendly, easy, simple, rapid, fast, efficient, several, state-of-the-art, improved, maximise, and minimise. Think what is the logical test that provides evidence of pass or fail.  Words that are clear to the requirements author may not be clear to readers. Write each requirement in clear, simple language, avoid techno-gabble – if this is necessary, then define it in a glossary.  To reveal ambiguity:  Carry out formal inspections of the requirements specifications,  Write test cases from requirements,  Create user scenarios illustrating any expected product behaviour.
  13. 13. 13 Checkable – Verifiable  Each requirement is properly implemented in the product. This is achieved by:  Tests which are repeatable with clear pass / fail criteria.  Verification approaches, such as inspection or demonstration.  If a requirement is not verifiable, then just a matter of opinion. The Requirement was of poor quality not the product.  Requirements must be consistent, feasible, unambiguous and verifiable.  Any requirement that says the product shall "support" something is not verifiable. (Support to what level? Be specific).
  14. 14. 14 On its Own – Single Logical Statements  A single Requirement will have a single test or group of tests associated to prove delivery.  A Change in requirement means a change in test.  Within the configuration control – build in triggers, otherwise testing may not test the latest requirements.  DO NOT write long winded paragraphs sighting great detail of functionality in one single Requirement clause. Avoid “AND”, “OR”, “Either”, etc. Single requirement – single proof (although perhaps many tests for a single proof set – e.g. boundary values).  Where necessary break top level requirements (A- Level) into lower (B-Level) requirements to provide clarity. Testing all B-levels will automatically mean that the A-level requirement needs no further testing in isolation (since all components were tested).
  15. 15. 15 Managing Success This section deals with how to manage requirements for a successful delivery.
  16. 16. 16 Review for Correctness Is the Requirement Correct? Is the requirement within the release scope? Is the requirement logically derived and not just an arbitrary value? Is the requirement technically possible? Is the requirement over-stringent? Is it a requirement (what) as opposed to a description of implementation (how)?
  17. 17. 17 Review for Ambiguity Is the Requirement Unambiguous and Measurable? A statement that can only be interpreted one way. Does not contain weak phrases or a poor sentence structure that can be misinterpreted.
  18. 18. 18 Review for Atomic, Traceable & Manageable  Is the Requirement Atomic, Traceable & Manageable?  Requirement is a single logical statement that cannot be separated out into ‘smaller’ sub-requirements.  Contains no logical AND/OR statements.  All requirements have a unique single identifier that can be used to map tests and software features to the requirements from which they were derived.  All lower sub-requirements are traceable to parent- requirements.  An infrastructure is in place to support the requirement and test traceability.  Related requirements are grouped together and refer to each other.  The requirement structure and style can be preserved even if the parameter value of the requirement change.
  19. 19. 19 Review for Testable & Verifiable  Is the Requirement Testable and Verifiable?  Can the requirement be tested with a pass/fail or quantitative assessment criteria derived from the specification itself and/or referenced information?  No subjective statements. Requiring that a system must be easy to use is subjective and therefore is not testable.  Are the valid / Invalid ranges defined?  Performance requirements state under what conditions the requirement is true. E.g. is a login response time to be measured for a single login, or while 200 virtual users are logging on spread over a 3 minute period?
  20. 20. 20 Review for Consistency Is the requirement set consistent? A consistent specification has no conflict between individual requirement statements. The defined behaviour of essential capabilities, specified behavioural properties and constraints do not have an adverse impact on that behaviour.  For example: “The only aircraft that is totally safe is one that cannot be started, contains no fuel or other liquids, and is securely tied down” (NASA).
  21. 21. 21 Review for Prioritisation Are Requirements Prioritised? Requirements need to be ranked according to importance for delivery, based upon the importance of the overall Business Goal.
  22. 22. 22 Review for Impact Assessment Are Requirements Impact Assessed? Is the requirement assessed for its impact [High(3)/Medium(2)/Low(1)] upon the business if the requirement fails?  NOTE: Test Risk Score = (Impact Score) x (Development Assessed Likelihood of Failure Score). The Development Assessed Likelihood of Failure Score can be assessed by the development team by assigning a numerical value: e.g.: Extremely Likely on a weekly bases (4), Likely on a quarterly bases (3), Possible (2), Outside small chance or lower (1).
  23. 23. 23 Managing review Feedback  To optimise the review feedback, consider maintaining a Review card to mark the requirement against each criteria (pass/fail).  The review should be carried out by users, the test manager and the development manager – so there is a full understanding and lack of future requirement blame. If a requirement later causes a problem then it is the fault of the review process.  Identify the mitigation actions required for all high risk requirements.
  24. 24. 24 Non Functional Requirements This section specifically looks at performance and security testing
  25. 25. 25 Special Case of Non Functional Requirements  13 non-functional requirements are specifically identified within IEEE-Std 830.  Performance  Security  Interface  Operational  Resource  Verification  Acceptance  Documentation  Portability  Quality  Reliability  Maintainability  Safety  Other groupings often identified include:  Implementation  Interface  Operations  Legal  Within this presentation 2 specifically are identified for special mention:  Performance  Security.
  26. 26. 26 Performance Testing Introduction This section starts to deal with performance testing. It is intended only to stimulate thought – where perhaps no previous consideration has been given to early performance testing. It does not cover the details of how to define and set up tests or in how to interpret results. It is purely focused on early mitigation.
  27. 27. 27 Performance Requirement Types Performance Criteria often include:  Availability - The proportion of time a system is in a functioning condition.  Response time - The interval from when an operator at a terminal enters a request, to the instant at which the first character of the response is received at a terminal. In data systems, the system response time is the interval from the receipt of an end of transmission for an inquiry message, to the start of a response transmission.  Channel (or Shannon) Capacity - The tightest upper bound on the rate of information that can be reliably transmitted over a communications channel.  Latency - The time delay experienced in a system  Completion time - The amount of time required for a specific task to be completed.  Service time - The time duration to service a specific item.  Bandwidth - The measurement of the bit-rate for available or consumed data communication, expressed in multiples of bits per second (bit/s, kbit/s, Mbit/s, Gbit/s, etc.) for computer networks.  Capacity - The total measure that can be contained or produced for the system. This can include for Example Requirement: data traffic or data storage.  Throughput - The average rate of successful message delivery over a communication channel.  Efficiency - The resource consumption for a given load.  Scalability - The ability for a system, network, or process to handle an increased amount of work in an appropriate manner, or the ability to be enlarged to accommodate growth or demand.  CPU benchmarks - The performance characteristics for a specific central processor unit. However this can apply to the running of software on a specific CPU type.  Resource constraints include processor speed, memory, disk space, network bandwidth, etc.
  28. 28. 28 General Principles for Performance Requirements 1. Avoid phrases such as “immediately” or “instantaneous” All transactions take some time. 2. Avoid requirements that would need control over external systems. For example to carry out a timing test over the internet would mean that the traffic would need to be controlled and kept consistent each time. So consider where the measurement is to be taken and under what conditions. A simulated load might be required and the user terminal might need to be connected at the data centre, for the timing to be valid. 3. Think about tolerances of readings. E.g. If a timing is to be measured as 3 seconds, would the test fail if the reading was either 2.99s or 3.2 sec? Specify either the top or bottom limit as appropriate. If necessary specify the range or tolerance permitted.
  29. 29. 29 Example Performance Requirements  The system shall be available 24/7 for 363 days in any one calendar year.  When an operator requests data, it shall start to be presented on the screen within no more than 3 seconds. Based upon a normal background scenario running (As described in document XYZ) and the user terminal being connected to the server.  The channel capacity for 6.2GHz at a distance of 11m shall be greater than 10-6.  for a web page download containing 512Kb of text, the latency shall not be greater than 900 ms.  Upon submitting a request to create a new user profile, the profile shall be created within 2.1 seconds (+/- 0.2s), based upon a normal background scenario running (As described in document XYZ).  The timing from the moment the credit transaction is submitted to the server, a user information message for the outcome of the transaction is to be sent to the user within no more than 10 seconds.  The interface shall transmit data at a minimum rate of 9,600 bits/second.  The spectral efficiency (or modulation efficiency) is to be not less than:  18.1 (bit/s)/Hz downstream.  15.5 (bit/s)/Hz upstream.  The telecommunications bandwidth shall be not less than 4.5 KHz.  Application X shall be able to be installed on a Windows 7 Pro system with a minimum of 2Gb of free hard drive space.  For an Ethernet connection of between 98 and 101 Mbit/s, the throughput shall not be less than 70 Mbit/s.  When dealing with current process demand levels (detailed within document ABC), the replacement system shall be at least 40% more energy efficient than the existing systems (whose energy consumption figures are contained within document ABC).  The system shall be scalable to accommodate growth from 1,000 records to 500 million records.  Game X shall run at 60 +/- 2 frames per second on a system containing a Corei7-2620M 2.7GHz processor with a HP Elitebook 8510w Nvidia 256mb laptop video graphic card.  Application X shall function with a minimum of 4GB of RAM on a laptop containing a Corei7-2620M 2.7GHz processor.
  30. 30. 30 General rule for testers  Do not leave performance till post integration. At that point it can be too late to fix problems and so only patching is possible, potentially opening up security weaknesses. Instead: 1. Create a realistic understanding of the operational conditions. 2. Design in performance – check the requirements and design. 3. Review code for performance issues. 4. Test unit code for performance. 5. Test the application for performance. 6. Test the system for performance.
  31. 31. 31 Security Section This section starts to deal with security testing. It is intended only to stimulate thought – where perhaps no previous consideration has been given to early security testing. It does not cover the details of security testing or penetration testing.
  32. 32. 32 Security Requirements  Security requirements seek to ensure:  Denial of unauthorised access to the system and the system data.  Defend the integrity of the system from malicious (including harmful accidental) damage  There are at least two basic metric groupings:  The ability to resist unauthorised attempts at usage  Continued service to genuine users while the system is under attack - avoiding Denial of Service (DoS).
  33. 33. 33 Negative and Non-Functional  Software is often not tested for security. At best the system may be subjected to a penetration test, which can often be a high level assessment. To improve security, software must be secure by design.  However usually requirements for software security are not specified. Further, negative and non-functional requirements may not be documented because they are not considered by requirements authors as “testable” or “actionable”.  To design in security from the start, the requirements author needs to go through a the process of listing the negative and non- functional requirements that are crucial to security.  Next the requirements author needs to analyse the negative and non-functional requirements to determine the resulting functional requirements required to support security.  The catch is the requirements authors may not have the skill set to do this.
  34. 34. 34 Early Security Consideration  When addressing security early consider:  Assume the system will be hacked. What is the inner defence?  Normal entry point may be secure by password, but what if it is disclosed, or data is stolen by an employee?  Do all users have access to all data?  Are there any back door connections to 3rd party systems that may be less secure?  For data entry are the fields open to any data – allowing SQL injection? Limit data content and size.  Is there any security monitoring in place – do not rely upon this – still test applications!
  35. 35. 35 Security Risk  In assessing security risk, the use of metrics is important, since that will direct the tester to the greatest need for mitigation.  Authentication metrics.  User types and access to data.  Resistance to known attack types.  Probability of finding a key – time, effort and resources needed.  Percentage of services available during an attack.  Lifespan of a password.  Lifespan of a session.
  36. 36. 36 Agile response to Non-functional testing  For the tester, the key to success is to ensure that non-functional testing starts early.  Within an Agile delivery approach, this means including non-functional requirement tests in the early burn-down lists, ensuring that test activities are included. The scope of the tests may well grow at each sprint, but they need to be included from the start.
  37. 37. 37 Mitigating Security Risk Understand the threats and assume the worse will happen. What is the next line of defence. Review the requirements and architecture for security. Review code early for security issues. Test the application for security issues. Test the system for security issues.
  38. 38. 38 END Slides created by: Ian R. McDonald 