Course contentsThe course consists of two parts:• Requirements – a description of what we shall develop: 11 lectures• Tests – a description of how we will check that we have developed what the customer required: 19 lecturesThe curriculum consists of• A set of slides• Copies of journal articles
Course gradeThe course grade will be decided based on• One group exercise on writing requirements – 10%• One group exercise on writing a test strategy for a set of requirements – 10%• One group exercise on testing a software package – 30%• Exam – 50%
Lecture plan – 1Tuesdays 14:15 – 16:00 in F3Fridays 10:15 – 11:00 in F3• First lecture – Tuesday, January 11, 2011.• Last lecture – Friday, April 15, 2011.• Exercise 1 – week 4.• Exercise 2 – week 7.• Exercise 3 – weeks 13 and 14.There will be no lectures during the exercise periods.
Lecture plan – 2Week Course content What you will learn Presentation of lecture plan and exercises 2 Goal oriented requirements specification Quality issues in requirements Requirements elicitation with boilerplates (templates) On requirement – testability and traceability – 1 3 On requirement – testability and traceability – 2 Intro. to exercise 1 4 Exercise 1 – Write a requirements specification Testing vs. inspection – strong and weak sides 5 Testing vs. inspection – strong and weak sides Testing and cost / benefit analysis Testing strategies – a general introduction Black box vs. White box 6 Grey box testing Introduction to exercise 2 7 Exercise 2 – Write a testing strategy for the requirements specification from exercise 1
How to prioritize requirements8 Requirements handling in COTS, Outsourcing, sub-contracting etc. Requirements in embedded and information systems Aspects, cross-cutting concerns and non-functional requirements9 Testing COTS, Outsourcing, sub-contracting etc. Presentation of the FITNESS tool Domain testing10 Coverage measures Use of test coverage measures for code and requirements Requirements through user stories and scenarios11 How to write good user stories and scenarios Advanced use cases Test driven development – TDD 112 Test driven development – TDD 2 Introduction to exercise 313 Exercise 314 Exercise 3 Regression testing15 Non-functional requirements – safety, user friendliness etc. Testing non-functional requirements
Main themeThe main message of this course is that requirements and tests are two sides of the same coin.• Requirements without tests will be ignored.• Tests without requirements are meaningless.
The need for traceabilityIt must be possible to trace – From test to requirement. Why do we need this test? – From requirement to test. • Where is this requirement tested? • Staus: How many of the tests for this requirement has been executed successfully?
Challenges in Requirements Engineering What is a requirement? •What a system must do (functional): System requirements •How well the system will perform its functions (non-functional): Institutt for datateknikk og System quality attributes informasjonsvitenskap The RE process: Inah Omoronyia and Tor StålhaneRequirements Specification and Testing An introduction defined satisfy Ultimately: operational business needs capabilities TDT 4242 TDT 4242 TDT 4242Challenges in Requirements Engineering Challenges in Requirements Engineering Importance of getting requirements right: 1/3 budget to correct errors originate from requirements Source: Benoy R Nair (IBS software services) TDT 4242 TDT 4242
Challenges in Requirements Engineering Challenges in Requirements Engineering Factors that make a software project challenging: Why projects are cancelled: Source: Benoy R Nair (IBS software services) Source: Benoy R Nair (IBS software services) TDT 4242 TDT 4242Requirements Development - 1 Requirements Development – 2 Requirements Elicitation: Effects of Inadequate Requirements development – Ariane 5: The process of discovering the requirements for a system by (An expendable launch system used to deliver payloads into communication with customers, system users and others who have a geostationary transfer orbit or low Earth orbit) stake in the system development. Ariane 5 succeeded Ariane 4. Wrong implicit assumptions about the Requirements gathering techniques parameters, in particular the horizontal velocity that were safe for Ariane 4 but not Ariane 5. • Methodical extraction of concrete requirements from high • horizontal velocity exceeded the maximum value for a 16 bit unsigned integer level goals when it was converted from its signed 64 bit representation. • Requirements quality metrics • Ariane 5: component (requirements) should have been designed for reuse – but the context of reuse was not specified. Cost of poor requirements in Ariane 5 • Data overflow on launch • Self-destruction of the complete system • Loss > 500 Million EUR TDT 4242 TDT 4242
Requirements Development – 3 Problem world and machine solution Effects of Inadequate Requirements development – Airbus: The problem to be solved is rooted in a complex organizational, technical or physical world. Requirement: Reverse thrust may only be used, when the airplane is • The aim of a software project is to improve the world by building some machine landed. expected to solve the problem. • Problem world and machine solution each have their own phenomena while sharing Translation: Reverse thrust may only be used while the wheels are others. rotating. • The shared phenomena defines the interface through which the machine interacts with the world. Implementation: Reverse thrust may only be used while the wheels are rotating fast enough. Situation: Rainstorm – aquaplaning Result: Crash due to overshooting the runway! Problem: erroneous modeling in the requirement phase E-commerce world Requirements engineering is concerned with the machine’s effect on the surrounding world and the assumption we make about that world. TDT 4242 TDT 4242 Formulation of requirements statements Two types of requirements statements • Descriptive statements: state properties about the system that holds regardless of how the system behaves. E.g. If train doors are open, they are not closed. • Prescriptive statements: States desirable properties about the system that may hold or not depending on how the system behaves • Need to be enforced by system componentsStatement scope: • E.g. Train doors shall always remain closed when the• Phenomenon of train physically moving is owned by environment. It train is movingcannot be directly observed by software phenomenon• The phenomenon of train measured speed being non-null is shared bysoftware and environment. It is measured by a speedometer in theenvironment and observed by the software. TDT 4242 TDT 4242
Formulation of system requirement Formulation of software requirementA prescriptive statement enforced by the software-to-be. A prescriptive statement enforced solely by the software-to-• Possibly in cooperation with other system components be. Formulated in terms of phenomena shared between the• Formulated in terms of environment phenomena software and environment.Example: The software “understand” or “sense” the environmentAll train doors shall always remain closed while the train is through input datamoving Example: In addition to the software-to-be we also requires the The doorState output variable shall always have the value cooperation of other components: ‘closed’ when the measuredSpeed input variable has a non- • Train controller being responsible for the safe control null value of doors. • The passenger refraining from opening doors unsafely • Door actuators working properly TDT 4242 TDT 4242Domain properties Goal orientation in requirements engineering – 1A domain property:• Is a descriptive statement about the problem world A goal is an objective that the system under• Should hold invariably regardless of how the system consideration shall achieve.behaves – Ranges from high-level strategic to low-level• Usually corresponds to some physical laws technical concerns over a system – System consist of both the software and itsExample: environment. Interaction between activeA train is moving if and only if its physical speed is non-null. components, i.e. devices, humans, software etc also called Agents TDT 4242 TDT 4242
Goal orientation in requirements engineering – 2 Goal statement typologyGoals can be stated at different levels of granularity: – High-level goal: A goal that requires the cooperation of many agents. They are normally stating strategic objective related to the business, e.g. The system’s transportation capacity shall be increased by 50% – Requirement: A goal under the responsibility of a single agent in the software-to-be. – Assumption (expectation): A goal under the responsibility of a single agent in the environment of the software-to-be. Assumptions cannot be enforced by the software-to-be TDT 4242 TDT 4242 Goal types Behavioral goal specialization TDT 4242 TDT 4242
Goal categorization – 1 Goal categorization – 2Goal categories are similar to requirements categories: Functional goal: States the intent underpinning a system service • Satisfaction: Functional goals concerned with satisfying agent request • Information: Functional goals concerned with keeping agents informed about important system states • Stimulus-response: Functional goals concerned with providing appropriate response to specific event Example: The on-board controller shall update the train’s acceleration to the commanded one immediately on receipt of an acceleration command from the station computer TDT 4242 TDT 4242Goal categorization – 3 Goal refinement A mechanism for structuring complex specifications at different levels ofNon-functional goal: States a quality or constraint on concern.service provision or development. A goal can be refined in a set of sub-goals that jointly contribute to it.Accuracy goal: Non-functional goals requiring the state of Each sub-goal is refined into finer-grained goals until we reach a requirement on the software and expectation (assumption) on thevariables controlled by the software to reflect the state of environment.corresponding quantities controlled by environment agent NB: Requirements on software are associated with a single agent and E.g: The train’s physical speed and commanded speed they are testable may never differ by more than X miles per hourSoft goals are different from non-functional goals. Soft goalsare goals with no clear-cut criteria to determine theirsatisfaction. E.g: The ATM interface should be more user friendly TDT 4242 TDT 4242
Goal refinement tree – 1Goal refinement: Example Refinement links are two way links: One showing goal decomposition, the other showing goal contribution TDT 4242 TDT 4242Goal refinement tree – 2Goal feature annotation Requirements quality metrics – 1 Qualitative Goal‐Requirements tracing: An approach to requirements refinement/abstraction that makes it less likely to generate trace links that are ambiguous, inconsistent, opaque, noisy, incomplete or with forward referencing items TDT 4242 TDT 4242
Requirements quality metrics – 2 Requirements quality metrics – 3 Forward Referencing: Requirement items that make use of problem world domain features that are not yet defined. E, C and D need to be mapped to a requirement item TDT 4242 TDT 4242Requirements quality metrics – 4 Requirements quality metrics – 5 Opacity: Requirement items for which rational or dependencies are invisible. Noise: Requirement items that yield no information on Multiple unrelated concept mapping. A is not related to B problem world features. X refers to a concept undefined in the domain TDT 4242 TDT 4242
Requirements quality metrics – 6 Requirements quality metrics Completeness: The needs of a prescribed Quality metrics on a requirements set provides useful system are fully covered by requirement understanding, tracking and control of requirements items without any undesirable outcome. improvement process. No requirement item mentions the goal concept Z TDT 4242Where do the goals come from? Summary We get goals from: Goals can be defined at different levels of • Preliminary analysis of the current system. abstraction • Systematically by searching intentional keywords There are two types of goals: Behavioral or soft goal in documents provided, interview transcripts etc. E.g. ‘objective’, ‘purpose’, ‘in order to’. • Iterative refinement and abstraction of high‐level There are several categories of goals, e.g. goals: By asking the how and why question. • Functional and non-functional Results in a goal refinement tree • Goal refinement provides a natural mechanism • Approaches: KAOS – Goal driven requirements for structuring complex specifications at acquisition. different levels of concern: • Goal refinement graph TDT 4242 TDT 4242
Requirements There are three levels of requirements: • Informal – e.g. Natural language (NL): free text, no rules apply Institutt for datateknikk og informasjonsvitenskap • Semiformal • Guided Natural Language (GNL): free text but Inah Omoronyia and Tor Stålhane allowable terms are defined by a vocabulare • Boilerplates (BP): structured text and an ontology – vocabulary plus relationships Guided Natural Language between terms and • Formal: e.g. state diagrams or predicate logic Requirement Boilerplates TDT 4242 TDT 4242 TDT 4242 Requirements elicitation Humans and machines – 1Step 1: Step 2: Step 3: Step 4: Given the amount and complexity of RE, we Capture Transfer Requirements and Refine the requirements model Create aRequirements in functions into a semi-formal and derive detailed requirements preliminary need to automate as much as possible.Natural Language requirements model design model based on the requirement model (to be Humans and machines have different strong Req.012: The systemshall enable cabin Function 1 Function 2 Function 1 used and and weak points.temperature regulation refined inbetween 15°C and 30°C … Req 001 Req 002 Req 011 Req 028 Function 1a Req 001.01 SP3) We want to elicit and analyze requirements in a Req 012 Req 050 …Req.124: Cabin … … Function 1b Req 001.02 …. way that allows both parties to build on their Req 124 …temperature shall notexceed 35° Function 1c strong sides. Parallel Steps: Apply dictionary with common vocabulary; validate and check Requirements consistency and completeness
Humans and machines - 2 Why BPs and GNL – 1 Machines are GNL and BPs will reduce variation and thus giving good at observing quantitative data and being the machines the opportunity to do what they are deductive, fast and precise. In addition, they are best at: to be fast, precise and consistent. good at doing consistent repetition of several By combining humans and machines and let both do actions. what they are best at, we get a better result than we bad at handling variations in written material and would get if we left the job of handling requirements pattern recognition. to just one of them. Humans are good at handling variations in written material, being inductive. In addition, they are good at doing error correction.Why BPs and GNL - 2 GNL and BPs Template based textual = Syntax + Semantics + Meta Model The final goal is to allow the machine to assist the Keywords: RMM - Refinement developers in analysing requirements for: Guided RSL Boilerplates Reflects requirement, system and domain Analysis - Specialization concepts -Correctness Consistency -Completeness Requirements expressed on templates -Consistency Completeness Uses predefined templates based on concepts, relations and axioms to guide requirements elicitation -Safety analysis Safety implications Example: The <system function> shall provide <system capability> to achieve <goal> Requirements expressed using a vocabulary guide Uses predefined concepts, relations and axioms to guide requirements elicitation Example: The ACC system shall be able to determine the speed of the ego-vehicle. Ontology: General and SP specific - Requirements classification - System attributes - Domain concepts
What is GNL - 1 What is GNL - 2 Aim: Free text requirement elicitation with the • Bridge the gap between unconstrained assistance of prescribed words from a expression and quality checking when dictionary. This will give us requirements representing requirements as free text. Quality which use all terms in a uniform way, this measures: reducing misunderstandings Correctness, consistency, completeness and un‐ No formal constraints ambiguity (reduced variability) Requires minimal expertise. • Provide the basis for semantic processing and checking of requirements. • Dictionary – Simple taxonomy or more formal ontologyApproach for GNL – 1 Approach for GNL – 2Ontology = Thesaurus + Inference Rules Required Activity • Thesaurus – Domain concepts: entities, Knowledge capture: Information embedded in domain events from domain experts and ontologist terms and events Implementation: Formal representation of captured • Inference Rules – Relations, attributes and knowledge. Language: OWL, Support environment: axioms Protégé. Verification: Checking that represented ontology is • Causality, similarity, reflexivity, correct using transitiveness, symmetric, disjoint • Classifiers/reasoners • Domain experts (semantic accuracy) (contradiction) … • Mapping of requirement segments to ontology concepts
Motivation for use of templates - 1 Motivation for use of templates - 1Text has the advantage of unconstrained Template based textual requirements expression. There is, however, a need for specification (boilerplates) will introduce some common limitations when representing requirements but will also reduce the opportunity to • Understanding of concepts used to express introduce ambiguities and inconsistencies. the requirements and relations between them. Boilerplates • Format of presentation • Provides an initial basis for requirementsLack of common understanding makes checking requirement specifications expressed as free text prone to ambiguous representations and • Are easy to understand for stakeholders inconsistencies. compared to more formal representations TDT 4242 TDT 4242What is a boilerplate – 1 What is a boilerplate – 2Boilerplates is a set of structures that can be used to The RE process is as follows: write requirements. They use high-level concept classification and attributes 1. Select a boilerplate or a sequence of boilerplates. The selection is based on the attributes that need to be included and how they are organized – fixed terms. 2. If needed, identify and include mode boilerplates 3. Instantiate all attributes A boilerplate consists of fixed terms and attributes. It may, or may not, contain one or more modes. TDT 4242 TDT 4242
Fixed Terms Boilerplate examples - 1Attributes BP32 The <user> shall be able to <capability> Attributes: • <user> = driver • <capability> = start the ACC system Requirement The driver shall be able to start the ACC system TDT 4242
Boilerplate examples - 2 Boilerplate examples - 3 BP43 While <operational condition> BP2 BP32 The <user> shall be able to <capability> The <system> shall be able to <action> <entity> BP43 is a mode Attributes: •<system> = ACC system Attributes •<action> = determine • <operational condition> = activated •<entity> = the speed of the ego-vehicle • <user> = driver • <capability> = override engine power control of Requirement the ACC system The ACC system shall be able to determine the Requirement speed of the ego-vehicle While activated the driver shall be able to override engine power control of the ACC-system TDT 4242 TDT 4242 Functional requirements example Non functional requirement example – 1Functional requirements from the SafeLoc system The robot control system shall stop the robot within 10 Non‐functional requirements and soft goals fits into milliseconds if a gate is opened to the zone where the robot is the same BPs as functional requirements operating The robot shall only be allowed to start when all gates are BP61 closed and the reset button is pushed The <system> shall be able to <action> to <entity> The robot shall stop if it tries to move into a zone already occupied by an operator Suitability: The <system > shall be able to <provide an appropriate set of functions> to <the user> TDT 4242 TDT 4242
Non functional requirement example – 2 Non functional requirement example – 3Non‐functional requirements and soft goals fits into the same BPs as functional requirements BP43 While <operational condition>BP2-1 The <system> shall be able to <capability> BP2BP12 …for a sustained period of at least The <system> shall be able to <action> <entity> <number> < unit> While <normal operational condition> theMaturity: <system> shall be able to <tolerate> The <system > shall be able to <operate without <90% of software faults of category...> failure> for a sustained period of at least <quantity> <time unit> TDT 4242Summing upThe use of boiler plates and ontologies will• Enforce a uniform use of terms• Reduce the variability of presentations – requirements that are similar will look similarReduced variation in form and contents simplifies the use of automatic and semi-automatic tools for• Checking requirement quality – e.g completeness and consistency• Creating test cases TDT 4242
What is requirements traceability “Requirements traceability refers to the ability to describe and follow the life of a Institutt for datateknikk og requirement, in both a forwards and informasjonsvitenskap backwards direction, i.e. from its origins, Inah Omoronyia and Tor Stålhane through its development and specification, to its subsequent deployment and use, and Requirements Traceability through periods of on‐going refinement and iteration in any of these phases.” Gotel and Finkelstein TDT 4242 TDT 4242 TDT 4242Traceability Goals - 1 Traceability Goals – 2 • Project Management • Validation – Status: “When will we finish?” and “what will it cost?” – finding and removing conflicts between requirements – Quality: “How close are we to our requirements?” – completeness of requirements • derived requirements cover higher level requirements • QA manager • each requirement is covered by part of the product – Improve Quality: “What can we do better?” • Verification • Change Management – assure that all requirements are fulfilled – Versioning, documentation of changes (Why? What? • System Inspection When?) – identify alternatives and compromises – Change Impact analysis • Certification/Audits • Reuse – proof of being compliant to standards – Variants and product families – Requirements can be targeted for reuse TDT 4242 TDT 4242
Habitat of Traceability Links – 1 Habitat of Traceability Links – 2 Pre- vs. Post-Requirements Specification TDT 4242 TDT 4242Challenges of traceability – 1 Challenges of traceability – 2 – Traces have to be identified and recorded among numerous, heterogeneous entity instances – A variety of tool support (document, models, code, . . . ). It is challenging • based on traceability matrix, hyperlink, tags to create meaningful relationships in such a and identifiers. complex context. • still manual with little automation – Traces are in a constant state of flux since they – Incomplete trace information is a reality due to may change whenever requirements or other complex trace acquisition and maintenance. development artefacts change. – Trust is a big issue: lack of quality attribute • There is no use of the information that 70% of trace links are accurate without knowing which of the links forms the 70% TDT 4242 TDT 4242
Challenges of traceability – 3 Challenges of traceability – 4 Different stakeholders usage viewpoint (different questions Different stakeholders usage viewpoint (different questions asked by different stakeholders): asked by different stakeholders): • Validation: • QA management: – Tracability can be used as a pointer to the quality of – “how close are we to our requirements” and “what can we requirements: do better” to improve quality. • Completeness, ambiguity, correctness/noise, • Change management inconsistency, forward referencing, opacity – Tracking down the effect of each change to each involved – Ensures that every requirement has been targeted by at component that might require adaptations to the change, least one part of the product recertification or just retesting to proof functionality. • Verification • Reuse: – Checking that constraints are not violated (in most cases – Pointing out those aspects of a reused component that this is an extension of validation context) need to be adapted to the new system requirements. • Certification/Audit – Even the requirements themselves can be targeted for reuse. • Testing, maintenance (reverse engineering) TDT 4242 TDT 4242Traceability meta-models – 1 Traceability meta-models – 2• A model is an abstraction of phenomena in the real world; a meta model is yet another abstraction, highlighting properties of the model itself.• Meta‐models for traceability are often used as the basis for the traceability methodologies and frameworks: – Define what type of artefacts should be traced. – Define what type of relations could be established between these artefacts. Traceability Meta Model Low-end traceability TDT 4242 TDT 4242
Traceability meta-models – 3 European EMPRESS project: Meta model for requirements traceabilityHigh-end traceability TDT 4242Traceability meta-models – 4 Approaches to traceability Creating trace links: – Critical tasks in requirements traceability: establish links between requirements and between requirements and other artefacts. – Manual linking and maintaining of such links is time consuming and error prone – Focus is on requirements traceability through (semi‐)automatic link generation. PRECISE Meta‐model (SINTEF)
Manual trace links – 1 Manual trace links – 2This is the classical traceability methods andthe simplest form of traceability. In this approach, we create a requirements traceability matrices using a hypertext or table cross referencing scheme, often using ExcelTwo problems • Long-term difficulty of maintaining a large numbers of links. • The static nature of the links (lack of attributes) limit the scope of potential automation.Scenario driven traceability – 1 Scenario driven traceability – 2 • Test‐based approach to uncover relations amongst The method to achieve traceablity uses the idea of requirements, design and code artifacts (Alexander “footprint”. Egyed ) When we are dealing with traceability, a footprint • Accomplished by observing the runtime behavior of contains two types of information: test scenarios. • The set of classes that were executed when we were • IBM Rational PureCoverage, open source tool testing a specific scenario. (org.jmonde.debug.Trace) • The number of methods that were executed in each • Translate this behavior into a graph structure to class. indicate commonalities among entities associated with the behavior
Footprints – 1 Footprints – 2E.g. scenario A uses 10 methods in class CAboutDlg Only classes are registered – e.g scenario [s3] uses and 3 methods in Csettings Dlg classes C, J, R and U Footprints – 3 Footprints – 4Some problems: Based on the footprint table, we can make a requirements-to-class trace table• There might be scenarios that do not cover any requirement – e.g. [s3]• There are scenarios that belong to several requirements, e.g. [s9]Such scenarios will get separate rows in the trace matrix and will be marked with an F (Fixed) or a P (Probable), depending on how sure we are that a certain class belongs to this scenario.
Footprints – 5 Development footprints - 1Each test scenario will leave a footprint. If we make A solution that enables the project to construct one test scenario per requirement, then we get one traceability information during development has footprint per requirement. been suggested by I. Omoronyia et al. The method requires that each developerWe can make the footprints more fine grained and • Always identifies which requirement – e.g. use case thus get more information by using methods or – he is currently working on code chunks instead for classes. • Only works at one use case at a timeThis will require more work but also more – better – traceability information. Development footprints - 2 Development footprints - 3The result will be similar to the scenario testing We can extract more info from the development footprint table. process in order to understand better what hasThe resulting table will show which documents, been going on in the project. The next slides shows classes etc. have been accessed during work on • Types of access: C – Create, U – Update and V – this particular requirement – e.g. use case. ViewMain problem: “false” accesses – e.g. a developer • Timeline – e.g. date or time looks at some of the code of another requirement • Person – who did what and, more important, who for info. will have expertise on what? Each line in the table will show
Development footprints - 4 Scenario driven traceability – 3 Problems: – Semi‐automated but requires a large amount of time from system engineers to iteratively identify a subset of test scenarios and how they related to requirement artifacts. – Requirements that are not related due to non matching execution paths might be related in some other form (e.g calling, data dependency, implementation pattern similarity, etc). Trace by tagging – 1 Trace by tagging – 2This method is easy to understand and simple to implement. The problem is that it depends on heavy human intervention.The principle is as follows:• Each requirement is given a tag, either manually or by the tool.• Each document, code chunk, etc. are marked with a tag which tells which requirement it belongs to
Trace by tagging – 3 Trace by tagging – 4There are several ways to create tags, e.g.: The quality of traceability through tagging will depend• Single level tags – e.g. R4. This gives a standard on that we remember to tag all relevant documents. trace matrix It is possible to check automatically that all• Multilevel tags – e.g. R4, R4.1 and R4.2 where R4 documents in the project database is tagged. is the top level requirement and R4.1 and R4.2 are It is, however, not possible to check that this tagging sub-requirements. This gives us more detailed is correct. trace information Conclusion• Requirements traceability is an important aspect of requirements management• Stakeholders have different traceability information needs• Traceability can be complex for not trivial projects• Traceability meta-models provide insight on the type of traceability information required for a project• There exist several automated approaches for requirements traceability. The strength is in a synergy of different automated approaches
Tor StålhaneRequirements Specification and Testing Requirements testability
Testability definitionAccording to ISO 9126, testability is defined as:Testability: The capability of the software product to enable modified software to be validated.NOTE - Values of this sub-characteristic may be altered by the modifications under consideration.
Testability concernsTestability touches upon two areas of concern:• How easy is it to test the implementation?• How test‐friendly is the requirement?These two concerns are not independent and need to be considered together. We will first look at it from the requirements side.
TestabilityThree basic ways to check that we have achieved our goals:• Executing a test. Give input, observe and check output. A test can be a – Black box test – White box test – Grey box test• Run experiments• Inspect the code and other artifactsUsually, we will include all of these activities in the term testing
When to use whatThe diagram on the next slide is a high level overview of when to use• T – Tests. Input / output. Involves the computer system and peripherals.• E – Experiments. Input / output but involves also the users.• I – Inspections. Evaluation based on documents.
Concrete requirements from high level goals T EE T I I T E E E TDT 4242
TestabilityIn order to be testable, a requirement needs to be stated in a precise way. For some requirements this is in place right from the start: When the ACC system is turned on, the “Active” light on the dashboard shall be turned on.In other cases we need to change a requirement to get a testable version. The system shall be easy to use.
Testability challenges - 1Some requirements are more difficult to test than others. Problems might rise due to:• Volume of tests needed, e.g. response time or storage capacity.• Type of event to be tested, e.g. error handling or safety mechanisms.• The required state of the system before testing, e.g. a rare failure state or a certain transaction history.
Testability challenges – 2We can test the requirements at any level. Theformulation of the test will depend on the level
Making a requirement testable – 1One way to make requirements testable is the “Design by Objective” method introduced by Tom Gilb.The method is simple in principle but is in some cases difficult to use. There are two problems• The resulting tests can in some be rather extensive and thus quite costly.• The method requires access to the system’s end users.
Making a requirement testable – 21. What do you mean by <requirement>? This will give us either (a) a testable requirement or (b) a set of testable and non-testable sub-requirements.2. In case (a) we are finished. In case (b) we will repeat question 1 for each non- testable sub-requirement
Making a requirement testable – 3Requirement: Reverse thrust may only be used, when the airplane is landed.The important questions are• “How do you define landed?”• Who should you ask – e.g. pilots, airplane construction engineers, or airplane designers?
Requirements for testability – 1 First and foremost:The customer needs to know what he wants and why he wants it. In some cases it is easier to test if the user actually has achieved his goal than to test that the system implements the requirement.Unfortunately, the “why”-part is usually not stated as part of a requirement.
Requirements for testability – 2Each requirement needs to be• Correct, i.e. without errors• Complete, i.e. has all possible situations been covered?• Consistent, i.e. not in disagreement with other requirements.• Clear, i.e. stated in a way that is easy to read and understand – e.g. using a commonly known notation.
Requirements for testability – 3Each requirement needs to be• Relevant, i.e. pertinent to the system’s purpose and at the right level of restrictiveness.• Feasible, i.e. possible to realize. If it is difficult to implement, is might also be difficult to test.• Traceable, i.e. it must be possible to relate it to one or more – Software components – Process steps
CompletenessAll possible situations must be covered.“If X then….”, “If Y then….” Must also consider what will happen “If neither X nor Y…”Automatic door opener – what is missing?If the door is closed and a person is detected then send signal Open_Door. If no person is detected after 10 sec., send signal Close_Door.
ConsistencyConsistency is a challenge since we, at least in the general case, need a complete overview of all requirements.In most cases, we can make do with checking all requirements that are related to the same event, function or parameter.
Clear – 1This is mainly a question of representation such as choice of• Diagram notation• Description language• Level of detailsWho shall understand the requirement?• Customers• Developers, including hired-in consultants• Testers
Clear – 2Simple example:Print the “accounts ledger” for all accountsThis requirement is perfectly clear for developers who are working in the banking business.Other developers might experience some problems.
RelevantTwo questions are important:• Do we really need this requirement?• Is it at the right level of strictness – i.e. not too strict and not too lax.Only the second question is important for the tester.• Too strict means more work for the developers• Too lax means more work for the tester.
FeasibleThis question is really related to the contract but we should also consider it here – can we really do this?Testers can contribute to the feasibility question by asking how it should be tested. This will help / force everybody involved to make the requirement more clear and thus improve on the requirement.Requirements that are difficult to tests are also usually difficult to implement – mainly because they are badly defined.
Some sound advice The following set of advices on requirements and testability are quoted from Ludwig Consulting Services, LLC.They are not a definition and not “the final words” on requirements testability. Instead, they should be used as a checklist. That one of the following rules are not obeyed does not mean that the requirement is wrong. It should, however, be reviewed for potential problems.
Modifying PhrasesWords and phrases that include: • as appropriate• if practical• as required• to the extent necessary / practical. Their meaning • is subject to interpretation • make the requirement optional Phrases like "at a minimum" only ensure the minimum, while "shall be considered" only requires the contractor to think about it.
Vague WordsVague words inject confusion. Examples of frequently used vague verbs are: • manage • track• handle • flag Information systems receive, store, calculate, report, and transmit data. Use words that express what the system must do. Requirement: The system shall process ABC data to the extent necessary to store it in an appropriate form for future access. Correction: The system shall edit ABC data.
Pronouns With No ReferenceExample: It shall be displayed. When this occurs, the writer is usually relying on a nearby requirement in the requirements document for the meaning of "it." As requirements are assigned for implementation, they are often reordered and regrouped, and the defining requirement is no longer nearby.
Passive VoiceRequirements should be written in active voice, which clearly shows X does or provides Y.Passive voice: Z shall be calculated. Active voice: the system shall calculate Z.
Negative RequirementsEverything outside the system is what the system does not do. Testing would have to continue forever to prove that the system does not do something. State what the system does. Substitute an active verb that expresses what the system must do. • Change "the system shall not allow X," to "the system shall prevent Y." • Use the prefix "un," such as: The system shall reject unauthorized users.
Assumptions and Comparisons – 1 The requirement "the system shall increase throughput by 15%" sounds testable, but isnt. The assumption is "over current system throughput." By comparing to another system, the meaning of the requirement changes when the other system changes
Assumptions and Comparisons – 2 An example, sometimes found in requests for proposals, is: "The system shall address the future needs of users." The writer is probably thinking ahead to after the contract is awarded. The requirement is meaningless because whenever it is read, it will point to the future. A requirement on change management included in the project management processes, would make more sense than making it a requirement for the system.
Indefinite PronounsIndefinite pronouns are “stand in” for unnamed people orthings, which makes their meaning subject to interpretation.Some of these may find their way into requirements:• All • Everybody• Another • Everyone• Any • Everything• Anybody • Few• Anything • Many • Each • Most• Either • Much • Every
The implementationWe will shortly look at four concerns related to implementation and testability:• Autonomy of the system under test• Observability of the testing progress• Re‐test efficiency • Test restartability
Autonomy – 1 How many other systems are needed to test this requirement? It is best if it can be tested using only the SUT and the autonomy and testability is successively reduced as the number of other, necessary systems increase. If the other systems needed are difficult to include at the present we will have to write more or less complex stubs.
Example – 1 Requirement: “If the door is closed and a person is detected then send signal Open_Door”• Sensors and actuators can be tested in the lab.• The system with a simulated actuator: simulate a “person detected” signal on the sensor and check if a Open_Door signal is sent to the actuator.
Example – 2 We can build a complete system – door, sensor, door motor and software system and test by• Letting persons approach the sensor• Check if the door opens – Early enough – Fast enough
Observability How easy is it to observe the • Progress of the test execution? This is important for tests that do not produce output – e.g. the requirement is only concerned with an internal state change or update of a database. • Results of the test? Important for tests where the output is dependent on an internal state or database content.
Re‐test effiencey Retest efficiency is concerned with how easy it is to perform the “test – check – change – re‐ test” cycle. This includes • Observability – observe the test result• Traceability – identify all tests related to the change
Test restartability This is mostly a question of check points in the code. How easy is it to • Stop the test temporarily• Study current state and output• Start the test again from – The point where it was stopped – Start
Final commentsThat a requirement is testable does not necessarily mean that it is easy to test.In order to have testable requirements it is important that• The testers are involved right from the start of the project. It is difficult to add testability later.• The tests are an integrated part of the requirement
The goals ‐ 1Consider the following goals for an adaptive cruse controller – ACC:When the ACC is active, vehicle speed shall be controlled automatically to maintain one of the following:• time gap to the forward vehicle • the set speed
The goals ‐ 2In case of several preceding vehicles, the preceding vehicle will be selected automaticallyThe ACC shall support a range of types• Type 1a: manual clutch operation• Type 1b: no manual clutch operation, no active brake control• Type 2a: manual clutch operation, active brake control• Type 2b: active brake control
Exercise requirements • Identify the combinations of sub‐goals that will jointly contribute to each of the goals• Build a refinement graph showing the low‐ level goals, requirements , assumptions, domain properties and associated agents
What we will cover• Part 1 – Introduction – Inspection processes – Testing processes • Part 2 – Tests and inspections – some data – Inspection as a social process – two experiments and some conclusions
Adams’ data – 2 The main information that you get from the table on the previous slide is that• Some defects are important because they will happen quite often.• Most defects are not important since they will happen seldom. How can we tell the difference?
Testing and inspection – 1 The important message here is that testing cannot always be done. In the first, important phases, we have nothing to execute and will thus always have to do some type of inspection. This might be considered one of the weaknesses of traditional software engineering over Agile development.
Testing and inspection – 2 In order to understand the main differences between testing and inspection, we should consider Fit’s list.Based on this, we will give a short discussion of the relative merits of testing and inspection.
Area of Man MachinecompetenceUnderstanding Good at handling variations in Bad at handling variations in written material written materialObserve General observations, Specialized, good at observing multifunctional quantitative data, bad at pattern recognitionReasoning Inductive, slow, imprecise but Deductive, fast, precise but good at error correction bad error correctionMemory Innovative, several access Copying, formal access mechanismsInformation Single channel, less than 10 Multi channel, severalhandling bits per second Megabits per secondConsistency Unreliable, get tired, depends Consistent repetition of several on learning actionsPower Low level, maximum ca. 150 High level over long periods watt of timeSpeed Slow – seconds Fast
Man vs. machine – 1 Good when we need the ability to • Handle variation• Be innovative and inductive• Recognize and handle patternsNot so good when we need the ability to• Do the same things over and over again in a consistent manner• Handle large amount of data
Man vs. machine – 2 In order to do the best job possible we need processes where we let each part• Do what they are best at: – Man is innovative – Machine handles large amounts of data • Support the other with their specialties. – Machine supports man by making large amounts of information available – Man support machine by providing it with innovative input
General considerations ‐ documentsArchitecture, system, sub‐system and component design plus pseudo code. Here we can only use inspections. Man will use experience and knowledge to identify possible problems Machine can support by identifying information – e.g. find all occurrences of a string.
General considerations – code (1)For executable code, we can use inspection, testing or a combination of both. The size and complexity – degree of dynamism – of the code will, to a large degree, decide our choice.Other important factors are the degree of experience with • The programming language• The algorithms used
General considerations – code (2)Simple code• Start with inspection – all code• Design and run testsComplex code• Start with inspection – focus on algorithm and logic• Decide test completeness criteria – we cannot test everything• Design and run tests
Inspections – 1 The term “inspection” is often used in a rather imprecise manner. We will look at three types of inspection:• Walkthrough• Informal inspection – also called informal review• Formal inspection – also called formal review or just inspectionThe first two types are usually project internal while the last one is used as a final acceptance activity for a document.
Inspections – 2For all types of inspections:• The quality of the results depends on the experience and knowledge of the participants. “Garbage in – Garbage out”• It might be a good idea to involve customer representatives.
The walkthrough process Walkthrough is a simple process – mostly used for early decisions for an activity. The document owner:1. Makes a rough sketch of the solution – architecture, algorithm etc. 2. Presents – explain – the sketch to whoever shows up. 3. Registers feedback – improvements.
Walkthrough – pros and consPros:• Easy and inexpensive. Needs no extra preparation.• Collect ideas at an early stage of development.Cons:• No commitment from the participants• May collect many loose or irrelevant ideas
The informal inspection process Planning Rules, checklists, proceduresProduct Change Individual Loggingdocument requests checking meeting
Informal inspections – pros and consPros:• Is simple and inexpensive to perform.• Can be used at all stages of development• Usually has a good cost / benefit ratio• Needs a minimum of planningCons:• No participant commitment • No process improvement
The formal inspection processThe formal inspection process described below is – with small variations – the most commonly used. The version shown on the following slides stem from T. Gilb and D. Graham. We recommend this process as the final acceptance process for all important documents
Formal inspection process overview Change requestsProduct document Planning Process improvements Rules, checklists, procedures Edit Individual and Walk‐ Kick‐off Logging follow‐ through checking meeting up
Distribution of resources TypicalActivity Range % value %Planning 3–5 4Kick-off 4–7 6Individual checking 20 – 30 25Logging 20 – 30 25Editing 15 – 30 20Process brainstorming 15 – 30 16Leader overhead, follow up, entry, 3–5 4exit
Initiating the inspection process• The inspection process starts with a “request for inspection” from the author to the QA responsible.• The QA responsible appoints an inspection leader.• First step is always to check that the document is fit for inspection.
PlanningImportant planning points are:• Who should participate in the inspections – Who is interested? – Who have time available for preparation and meetings? – Who has the necessary knowledge concerning application, language, tools, methods?
Kick‐offImportant activities here are:• Distribution of necessary documents: – Documents that shall be inspected – Requirements – Applicable standards and checklists• Assignment of roles and jobs• Setting targets for resources, deadlines etc.
Individual checkingThis is the main activity of the inspection. Each participant read the document to look for • Potential errors ‐ inconsistencies with requirements or common application experience• Lack of adherence to company standards or good workmanship
Logging meetingThe logging meeting has three purposes:• Log issues already discovered by inspection participants• Discover new issues based on discussions and new information that arises during the logging meeting.• Identify possible improvement to the inspection or development process.
Improve the product ‐ 1The author receives the log from the inspection meeting. All items ‐ issues ‐ in the log are categorised as one of the following:• Errors in the author’s document.• Errors in someone else’s document.• Misunderstandings in the inspection team.
Improve the product ‐ 2• Errors in own document: Make appropriate corrections• Errors in someone else’s documents: Inform the owner of this document.• Misunderstandings in the inspection team: Improve document to avoid further misunderstandings.
Checking the changesThis is the responsibility of the inspection leader. He must assure that all issues raised in the log are disposed of in a satisfactory manner:• The documents that have been inspected• Related documents ‐ including standards and checklists• Suggested process improvements
Formal inspection – pros and consPros:• Can be used to formally accept documents• Includes process improvement Cons:• Is time consuming and expensive• Needs extensive planning in order to succeed
Testing We will look at three types of testing:• Unit testing – does the code behave as intended. Usually done by the developer• Function verification testing – also called systems test. Does the component or system provide the required functionality?• System verification testing – also called acceptance test. Does the hardware and software work together to give the user the intended functionality?
The unit testing processUnit testing is done by the developer one or more times during development. It is a rather informal process which mostly run as follows:1.Implement (part of) a component.2.Define one or more tests to activate the code3.Check the results against expectations and current understanding of the component
Unit testing – pros and consPros:• Simple way to check that the code works.• Can be used together with coding in an iterative manner.Cons:• Will only test the developer’s understanding of the spec.• May need stubs or drivers in order to test
The system test processA systems test has the following steps:1. Based on the requirements, identify – Test for each requirement, including error handling – Initial state, expected result and final state2. Identify dependencies between tests3. Identify acceptance criteria for test suite 4. Run tests and check results against – Acceptance criteria for each test – Acceptance criteria for the test suite
Systems test – pros and consPros:• Tests system’s behavior against customer requirements.Cons:• It is a black box test. If we find an error, the systems test must be followed by extensive debugging
The acceptance test processThe acceptance test usually has three actvities – both involving the customer or his representatives:• Rerun the systems test at the customer’s site.• Use the system to solve a set of real‐world tasks.• Try to break the system – by stressing it or by feeding it large amounts of illegal input
Acceptance test – pros and consPros:• Creates confidence that the system will be useful for the customer• Shows the system’s ability to operate in the customer’s environmentCons:• Might force the system to handle input that it was not designed for, thus creating an unfavorable impression.
Test and inspections – some termsFirst we need to understand two important terms – defect types and triggers.After this we will look at inspection data and test data from three activity types, organized according to type of defect and trigger. We need the defect categories to compare test and inspections – where is what best?
Defect categoriesThis presentation uses eight defect categories:• Wrong or missing assignment• Wrong or missing data validation• Error in algorithm – no design change is necessary• Wrong timing or sequencing• Interface problems• Functional error – design change is needed• Build, package or merge problem• Documentation problem
Triggers We will use different triggers for test and inspections. In addition – white box and black box tests will use different triggers. We will get back to triggers and black box / white box testing later in the course.
Inspection triggers• Design conformance• Understanding details – Operation and semantics – Side effects – Concurrency• Backward compatibility – earlier versions of this system• Lateral compatibility – other, similar systems• Rare situations• Document consistency and completeness• Language dependencies
Test triggers – black box • Test coverage• Sequencing – two code chunks in sequence• Interaction – two code chunks in parallel• Data variation – variations over a simple test case• Side effects – unanticipated effects of a simple test case
Test triggers – white box• Simple path coverage• Combinational path coverage – same path covered several times but with different inputs• Side effect ‐ unanticipated effects of a simple path coverage
Inspection dataWe will look at inspection data from three development activities:• High level design: architectural design • Low level design: design of subsystems, components – modules – and data models• Implementation: realization, writing code This is the left hand side of the V‐model
Test data We will look at test data from three development activities:• Unit testing: testing a small unit like a method or a class• Function verification testing: functional testing of a component, a system or a subsystem• System verification testing: testing the total system, including hardware and users. This is the right hand side of the V‐model
What did we find The next tables will, for each of the assigned development activities, show the following information:• Development activity• The three most efficient triggersFirst for inspection and then for testing
Inspection – defect typesActivity Defect type Percentage Documentation 45.10High level design Function 24.71 Interface 14.12 Algorithm 20.72Low level design Function 21.17 Documentation 20.27 Algorithm 21.62Code inspection Documentation 17.42 Function 15.92
Testing – triggers and defects Activity Trigger Percentage Test sequencing 41.90Implementation Test coverage 33.20testing Side effects 11.07Activity Defect type Percentage Interface 39.13Implementation Assignments 17.79testing Build / Package / 14.62 Merge
Some observations – 1 • Pareto’s rule will apply in most cases – both for defect types and triggers• Defects related to documentation and functions taken together are the most commonly found defect types in inspection – HLD: 69.81% – LLD: 41.44% – Code: 33.34%
Some observations – 2 • The only defect type that is among the top three both for testing and inspection is “Interface” – Inspection ‐ HLD: 14.12% – Testing: 39.13% • The only trigger that is among the top three both for testing and inspection is “Side effects” – Inspection – LLD: 29.73 – Testing: 11.07
Summary Testing and inspection are different activities. By and large, they• Need different triggers• Use different mind sets• Find different types of defects Thus, we need both activities in order to get a high quality product