TDT 4242Tor Stålhane
Course contentsThe course consists of two parts:• Requirements – a description of what we   shall develop: 11 lectures• Te...
Course gradeThe course grade will be decided based on• One group exercise on writing  requirements – 10%• One group exerci...
Lecture plan – 1Tuesdays 14:15 – 16:00 in F3Fridays 10:15 – 11:00 in F3• First lecture – Tuesday, January 11, 2011.• Last ...
Lecture plan – 2Week   Course content       What you will learn       Presentation of lecture plan and exercises 2     Goa...
How to prioritize requirements8    Requirements handling in COTS, Outsourcing, sub-contracting etc.     Requirements in em...
Main themeThe main message of this course is that  requirements and tests are two sides of  the same coin.• Requirements w...
The need for traceabilityIt must be possible to trace  – From test to requirement. Why do we need    this test?  – From re...
Challenges in Requirements Engineering                                                                    What is a requir...
Challenges in Requirements Engineering                                                  Challenges in Requirements Enginee...
Requirements Development – 3                                              Problem world and machine solution     Effects o...
Formulation of system requirement                                    Formulation of software requirementA prescriptive sta...
Goal orientation in requirements engineering – 2                Goal statement typologyGoals can be stated at different le...
Goal categorization – 1                                              Goal categorization – 2Goal categories are similar to...
Goal refinement tree – 1Goal refinement: Example                                        Refinement links are two way links...
Requirements quality metrics – 2                                  Requirements quality metrics – 3                        ...
Requirements quality metrics – 6                              Requirements quality metrics   Completeness: The needs of a ...
Requirements                                                                                                              ...
Humans and machines - 2                                          Why BPs and GNL – 1   Machines are                      ...
What is GNL - 1                                      What is GNL - 2                                                     A...
Motivation for use of templates - 1                                  Motivation for use of templates - 1Text has the advan...
Fixed Terms             Boilerplate examples - 1Attributes              BP32              The <user> shall be able to <cap...
Boilerplate examples - 2                                                 Boilerplate examples - 3                         ...
Non functional requirement example – 2                       Non functional requirement example – 3Non‐functional requirem...
What is requirements traceability                                                                                “Requirem...
Habitat of Traceability Links – 1                            Habitat of Traceability Links – 2                            ...
Challenges of traceability – 3                                                 Challenges of traceability – 4 Different st...
Traceability meta-models – 3                                         European EMPRESS project: Meta model for             ...
Manual trace links – 1                                     Manual trace links – 2This is the classical traceability method...
Footprints – 1                                         Footprints – 2E.g. scenario A uses 10 methods in class CAboutDlg   ...
Footprints – 5                                          Development footprints - 1Each test scenario will leave a footprin...
Development footprints - 4                             Scenario driven traceability – 3                                   ...
Trace by tagging – 3                                           Trace by tagging – 4There are several ways to create tags, ...
Tor StålhaneRequirements Specification and Testing        Requirements testability
Testability definitionAccording to ISO 9126, testability is defined as:Testability: The capability of the software product...
Testability concernsTestability touches upon two areas of concern:• How easy is it to test the implementation?• How test‐f...
TestabilityThree basic ways to check that we have achieved our  goals:• Executing a test. Give input, observe and check  o...
When to use whatThe diagram on the next slide is a high level  overview of when to use• T – Tests. Input / output. Involve...
Concrete requirements from high level goals        T                                       EE               T             ...
TestabilityIn order to be testable, a requirement needs  to be stated in a precise way. For some  requirements this is in ...
Testability challenges - 1Some requirements are more difficult to test  than others. Problems might rise due to:• Volume o...
Testability challenges – 2We can test the requirements at any level. Theformulation of the test will depend on the level
Making a requirement testable – 1One way to make requirements testable is  the “Design by Objective” method  introduced by...
Making a requirement testable – 21. What do you mean by <requirement>?   This will give us either (a) a testable   require...
Making a requirement testable – 3Requirement: Reverse thrust may only be used, when the airplane is landed.The important q...
Requirements for testability – 1  First and foremost:The customer needs to know what he wants  and why he wants it. In som...
Requirements for testability – 2Each requirement needs to be• Correct, i.e. without errors• Complete, i.e. has all possibl...
Requirements for testability – 3Each requirement needs to be• Relevant, i.e. pertinent to the system’s  purpose and at the...
CompletenessAll possible situations must be covered.“If X then….”, “If Y then….” Must also   consider what will happen “If...
ConsistencyConsistency is a challenge since we, at least in the general case, need a complete overview of all requirements...
Clear – 1This is mainly a question of representation  such as choice of• Diagram notation• Description language• Level of ...
Clear – 2Simple example:Print the “accounts ledger” for all accountsThis requirement is perfectly clear for developers who...
RelevantTwo questions are important:• Do we really need this requirement?• Is it at the right level of strictness – i.e. n...
FeasibleThis question is really related to the contract but  we should also consider it here – can we really  do this?Test...
Some sound advice The following set of advices on requirements and   testability are quoted from Ludwig Consulting   Servi...
Modifying PhrasesWords and phrases that include: • as appropriate• if practical• as required• to the extent necessary / pr...
Vague WordsVague words inject confusion. Examples of frequently   used vague verbs are: • manage • track• handle • flag In...
Pronouns With No ReferenceExample: It shall be displayed. When this occurs, the writer is usually relying on   a nearby re...
Passive VoiceRequirements should be written in active voice,   which clearly shows X does or provides Y.Passive voice: Z s...
Negative RequirementsEverything outside the system is what the system   does not do. Testing would have to continue foreve...
Assumptions and Comparisons – 1 The requirement "the system shall increase   throughput by 15%" sounds testable, but isnt....
Assumptions and Comparisons – 2 An example, sometimes found in requests for   proposals, is: "The system shall address the...
Indefinite PronounsIndefinite pronouns are “stand in” for unnamed people orthings, which makes their meaning subject to in...
The implementationWe will shortly look at four concerns related to   implementation and testability:• Autonomy of the syst...
Autonomy – 1  How many other systems are needed to test this    requirement? It is best if it can be tested using only the...
Example – 1  Requirement: “If the door is closed and a  person is detected then send signal  Open_Door”• Sensors and actua...
Example – 2 We can build a complete system – door, sensor,   door motor and software system and test by• Letting persons a...
Observability How easy is it to observe the • Progress of the test execution?  This is important for tests that do not pro...
Re‐test effiencey Retest efficiency is concerned with how easy it is   to perform the “test – check – change – re‐  test” ...
Test restartability This is mostly a question of check points in the   code. How easy is it to • Stop the test temporarily...
Final commentsThat a requirement is testable does not  necessarily mean that it is easy to test.In order to have testable ...
Introduction to exercise 1        Tor Stålhane
The goals ‐ 1Consider the following goals for an adaptive   cruse controller – ACC:When the ACC is active, vehicle speed s...
The goals ‐ 2In case of several preceding vehicles, the   preceding vehicle will be selected   automaticallyThe ACC shall ...
Exercise requirements • Identify the combinations of sub‐goals that   will jointly contribute to each of the goals• Build ...
ACC state model
Test vs. inspection      Part 1     Tor Stålhane
What we will cover• Part 1   – Introduction   – Inspection processes  – Testing processes • Part 2  – Tests and inspection...
Introduction 
Adam’s data ‐ 1                Mean Time to Problem Occurrence – yearsProduct   1.6     5      16    50    160    500    1...
Adams’ data – 2  The main information that you get from the   table on the previous slide is that• Some defects are import...
Testing and inspection – the V model
Testing and inspection – 1 The important message here is that testing   cannot always be done. In the first, important pha...
Testing and inspection – 2  In order to understand the main differences   between testing and inspection, we should   cons...
Area of         Man                              MachinecompetenceUnderstanding   Good at handling variations in   Bad at ...
Man vs. machine – 1  Good when we need the ability to • Handle variation• Be innovative and inductive• Recognize and handl...
Man vs. machine – 2  In order to do the best job possible we need   processes where we let each part• Do what they are bes...
General considerations ‐ documentsArchitecture, system, sub‐system and   component design plus pseudo code. Here we   can ...
General considerations – code (1)For executable code, we can use inspection,   testing or a combination of both. The size ...
General considerations – code (2)Simple code• Start with inspection – all code• Design and run testsComplex code• Start wi...
Inspection processes
Inspections – 1  The term “inspection” is often used in a rather   imprecise manner. We will look at three types   of insp...
Inspections – 2For all types of inspections:• The quality of the results depends on the   experience and knowledge of the ...
The walkthrough process Walkthrough is a simple process – mostly used    for early decisions for an activity. The    docum...
Walkthrough – pros and consPros:• Easy and inexpensive. Needs no extra   preparation.• Collect ideas at an early stage of ...
The informal inspection process             Planning                            Rules,                            checklis...
Informal inspections – pros and consPros:• Is simple and inexpensive to perform.• Can be used at all stages of development...
The formal inspection processThe formal inspection process described below   is – with small variations – the most   commo...
Formal inspection process overview                                                                            Change      ...
Distribution of resources                                            TypicalActivity                           Range %    ...
Initiating the inspection process• The inspection process starts with a “request   for inspection” from the author to the ...
PlanningImportant planning points are:• Who should participate in the inspections  – Who is interested?  – Who have time a...
Kick‐offImportant activities here are:• Distribution of necessary documents:  – Documents that shall be inspected  – Requi...
Individual checkingThis is the main activity of the inspection. Each   participant read the document to look for  • Potent...
Logging meetingThe logging meeting has three purposes:• Log issues already discovered by inspection   participants• Discov...
Improve the product ‐ 1The author receives the log from the inspection   meeting. All items ‐ issues ‐ in the log are   ca...
Improve the product ‐ 2• Errors in own document:  Make appropriate corrections• Errors in someone else’s documents:  Infor...
Checking the changesThis is the responsibility of the inspection   leader. He must assure that all issues raised   in the ...
Formal inspection – pros and consPros:• Can be used to formally accept documents• Includes process improvement Cons:• Is t...
Testing processes 
Testing We will look at three types of testing:• Unit testing – does the code behave as   intended. Usually done by the de...
The unit testing processUnit testing is done by the developer one or   more times during development. It is a rather   inf...
Unit testing – pros and consPros:• Simple way to check that the code works.• Can be used together with coding in an   iter...
The system test processA systems test has the following steps:1. Based on the requirements, identify  –   Test for each re...
Systems test – pros and consPros:• Tests system’s behavior against customer   requirements.Cons:• It is a black box test. ...
The acceptance test processThe acceptance test usually has three actvities –  both involving the customer or his   represe...
Acceptance test – pros and consPros:• Creates confidence that the system will be   useful for the customer• Shows the syst...
Test vs. inspection      Part 2    Tor Stålhane
Testing and inspection A short data analysis 
Test and inspections – some termsFirst we need to understand two important   terms – defect types and triggers.After this ...
Defect categoriesThis presentation uses eight defect categories:• Wrong or missing assignment• Wrong or missing data valid...
Triggers We will use different triggers for test and  inspections. In addition – white box and black  box tests will use d...
Inspection triggers• Design conformance• Understanding details   – Operation and semantics   – Side effects   – Concurrenc...
Test triggers – black box • Test coverage• Sequencing – two code chunks in sequence• Interaction – two code chunks in para...
Test triggers – white box• Simple path coverage• Combinational path coverage – same path   covered several times but with ...
Testing and inspection – the V model
Inspection dataWe will look at inspection data from three   development activities:• High level design: architectural desi...
Test data We will look at test data from three   development activities:• Unit testing: testing a small unit like a method...
What did we find The next tables will, for each of the assigned   development activities, show the following   information...
Inspection – defect typesActivity            Defect type     Percentage                    Documentation      45.10High le...
Inspection – triggers Activity            Trigger                Percentage                    Understand details       34...
Testing – triggers and defects Activity         Trigger             Percentage                 Test sequencing       41.90...
Some observations – 1  • Pareto’s rule will apply in most cases – both   for defect types and triggers• Defects related to...
Some observations – 2 • The only defect type that is among the top   three both for testing and inspection is   “Interface...
Summary Testing and inspection are different activities.   By and large, they• Need different triggers• Use different mind...
Inspection as a social process 
Inspection as a social processInspections is a people‐intensive process. Thus,   we cannot consider only technical details...
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Tdt4242
Upcoming SlideShare
Loading in...5
×

Tdt4242

1,013

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,013
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
20
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Tdt4242

  1. 1. TDT 4242Tor Stålhane
  2. 2. Course contentsThe course consists of two parts:• Requirements – a description of what we shall develop: 11 lectures• Tests – a description of how we will check that we have developed what the customer required: 19 lecturesThe curriculum consists of• A set of slides• Copies of journal articles
  3. 3. Course gradeThe course grade will be decided based on• One group exercise on writing requirements – 10%• One group exercise on writing a test strategy for a set of requirements – 10%• One group exercise on testing a software package – 30%• Exam – 50%
  4. 4. Lecture plan – 1Tuesdays 14:15 – 16:00 in F3Fridays 10:15 – 11:00 in F3• First lecture – Tuesday, January 11, 2011.• Last lecture – Friday, April 15, 2011.• Exercise 1 – week 4.• Exercise 2 – week 7.• Exercise 3 – weeks 13 and 14.There will be no lectures during the exercise periods.
  5. 5. Lecture plan – 2Week Course content What you will learn Presentation of lecture plan and exercises 2 Goal oriented requirements specification Quality issues in requirements Requirements elicitation with boilerplates (templates) On requirement – testability and traceability – 1 3 On requirement – testability and traceability – 2 Intro. to exercise 1 4 Exercise 1 – Write a requirements specification Testing vs. inspection – strong and weak sides 5 Testing vs. inspection – strong and weak sides Testing and cost / benefit analysis Testing strategies – a general introduction Black box vs. White box 6 Grey box testing Introduction to exercise 2 7 Exercise 2 – Write a testing strategy for the requirements specification from exercise 1
  6. 6. How to prioritize requirements8 Requirements handling in COTS, Outsourcing, sub-contracting etc. Requirements in embedded and information systems Aspects, cross-cutting concerns and non-functional requirements9 Testing COTS, Outsourcing, sub-contracting etc. Presentation of the FITNESS tool Domain testing10 Coverage measures Use of test coverage measures for code and requirements Requirements through user stories and scenarios11 How to write good user stories and scenarios Advanced use cases Test driven development – TDD 112 Test driven development – TDD 2 Introduction to exercise 313 Exercise 314 Exercise 3 Regression testing15 Non-functional requirements – safety, user friendliness etc. Testing non-functional requirements
  7. 7. Main themeThe main message of this course is that requirements and tests are two sides of the same coin.• Requirements without tests will be ignored.• Tests without requirements are meaningless.
  8. 8. The need for traceabilityIt must be possible to trace – From test to requirement. Why do we need this test? – From requirement to test. • Where is this requirement tested? • Staus: How many of the tests for this requirement has been executed successfully?
  9. 9. Challenges in Requirements Engineering What is a requirement? •What a system must do (functional): System requirements •How well the system will perform its functions (non-functional): Institutt for datateknikk og System quality attributes informasjonsvitenskap The RE process: Inah Omoronyia and Tor StålhaneRequirements Specification and Testing An introduction defined satisfy Ultimately: operational business needs capabilities TDT 4242 TDT 4242 TDT 4242Challenges in Requirements Engineering Challenges in Requirements Engineering Importance of getting requirements right: 1/3 budget to correct errors originate from requirements Source: Benoy R Nair (IBS software services) TDT 4242 TDT 4242
  10. 10. Challenges in Requirements Engineering Challenges in Requirements Engineering Factors that make a software project challenging: Why projects are cancelled: Source: Benoy R Nair (IBS software services) Source: Benoy R Nair (IBS software services) TDT 4242 TDT 4242Requirements Development - 1 Requirements Development – 2 Requirements Elicitation: Effects of Inadequate Requirements development – Ariane 5: The process of discovering the requirements for a system by (An expendable launch system used to deliver payloads into communication with customers, system users and others who have a geostationary transfer orbit or low Earth orbit) stake in the system development. Ariane 5 succeeded Ariane 4. Wrong implicit assumptions about the Requirements gathering techniques parameters, in particular the horizontal velocity that were safe for Ariane 4 but not Ariane 5. • Methodical extraction of concrete requirements from high • horizontal velocity exceeded the maximum value for a 16 bit unsigned integer level goals when it was converted from its signed 64 bit representation. • Requirements quality metrics • Ariane 5: component (requirements) should have been designed for reuse – but the context of reuse was not specified. Cost of poor requirements in Ariane 5 • Data overflow on launch • Self-destruction of the complete system • Loss > 500 Million EUR TDT 4242 TDT 4242
  11. 11. Requirements Development – 3 Problem world and machine solution Effects of Inadequate Requirements development – Airbus: The problem to be solved is rooted in a complex organizational, technical or physical world. Requirement: Reverse thrust may only be used, when the airplane is • The aim of a software project is to improve the world by building some machine landed. expected to solve the problem. • Problem world and machine solution each have their own phenomena while sharing Translation: Reverse thrust may only be used while the wheels are others. rotating. • The shared phenomena defines the interface through which the machine interacts with the world. Implementation: Reverse thrust may only be used while the wheels are rotating fast enough. Situation: Rainstorm – aquaplaning Result: Crash due to overshooting the runway! Problem: erroneous modeling in the requirement phase E-commerce world Requirements engineering is concerned with the machine’s effect on the surrounding world and the assumption we make about that world. TDT 4242 TDT 4242 Formulation of requirements statements Two types of requirements statements • Descriptive statements: state properties about the system that holds regardless of how the system behaves. E.g. If train doors are open, they are not closed. • Prescriptive statements: States desirable properties about the system that may hold or not depending on how the system behaves • Need to be enforced by system componentsStatement scope: • E.g. Train doors shall always remain closed when the• Phenomenon of train physically moving is owned by environment. It train is movingcannot be directly observed by software phenomenon• The phenomenon of train measured speed being non-null is shared bysoftware and environment. It is measured by a speedometer in theenvironment and observed by the software. TDT 4242 TDT 4242
  12. 12. Formulation of system requirement Formulation of software requirementA prescriptive statement enforced by the software-to-be. A prescriptive statement enforced solely by the software-to-• Possibly in cooperation with other system components be. Formulated in terms of phenomena shared between the• Formulated in terms of environment phenomena software and environment.Example: The software “understand” or “sense” the environmentAll train doors shall always remain closed while the train is through input datamoving Example: In addition to the software-to-be we also requires the The doorState output variable shall always have the value cooperation of other components: ‘closed’ when the measuredSpeed input variable has a non- • Train controller being responsible for the safe control null value of doors. • The passenger refraining from opening doors unsafely • Door actuators working properly TDT 4242 TDT 4242Domain properties Goal orientation in requirements engineering – 1A domain property:• Is a descriptive statement about the problem world A goal is an objective that the system under• Should hold invariably regardless of how the system consideration shall achieve.behaves – Ranges from high-level strategic to low-level• Usually corresponds to some physical laws technical concerns over a system – System consist of both the software and itsExample: environment. Interaction between activeA train is moving if and only if its physical speed is non-null. components, i.e. devices, humans, software etc also called Agents TDT 4242 TDT 4242
  13. 13. Goal orientation in requirements engineering – 2 Goal statement typologyGoals can be stated at different levels of granularity: – High-level goal: A goal that requires the cooperation of many agents. They are normally stating strategic objective related to the business, e.g. The system’s transportation capacity shall be increased by 50% – Requirement: A goal under the responsibility of a single agent in the software-to-be. – Assumption (expectation): A goal under the responsibility of a single agent in the environment of the software-to-be. Assumptions cannot be enforced by the software-to-be TDT 4242 TDT 4242 Goal types Behavioral goal specialization TDT 4242 TDT 4242
  14. 14. Goal categorization – 1 Goal categorization – 2Goal categories are similar to requirements categories: Functional goal: States the intent underpinning a system service • Satisfaction: Functional goals concerned with satisfying agent request • Information: Functional goals concerned with keeping agents informed about important system states • Stimulus-response: Functional goals concerned with providing appropriate response to specific event Example: The on-board controller shall update the train’s acceleration to the commanded one immediately on receipt of an acceleration command from the station computer TDT 4242 TDT 4242Goal categorization – 3 Goal refinement A mechanism for structuring complex specifications at different levels ofNon-functional goal: States a quality or constraint on concern.service provision or development. A goal can be refined in a set of sub-goals that jointly contribute to it.Accuracy goal: Non-functional goals requiring the state of Each sub-goal is refined into finer-grained goals until we reach a requirement on the software and expectation (assumption) on thevariables controlled by the software to reflect the state of environment.corresponding quantities controlled by environment agent NB: Requirements on software are associated with a single agent and E.g: The train’s physical speed and commanded speed they are testable may never differ by more than X miles per hourSoft goals are different from non-functional goals. Soft goalsare goals with no clear-cut criteria to determine theirsatisfaction. E.g: The ATM interface should be more user friendly TDT 4242 TDT 4242
  15. 15. Goal refinement tree – 1Goal refinement: Example Refinement links are two way links: One showing goal decomposition, the other showing goal contribution TDT 4242 TDT 4242Goal refinement tree – 2Goal feature annotation Requirements quality metrics – 1  Qualitative Goal‐Requirements tracing: An approach to requirements  refinement/abstraction that makes it  less likely to generate trace links that  are ambiguous, inconsistent, opaque,  noisy, incomplete or with forward  referencing items TDT 4242 TDT 4242
  16. 16. Requirements quality metrics – 2 Requirements quality metrics – 3 Forward Referencing: Requirement items that make use of problem world domain features that are not yet defined. E, C and D need to be mapped to a requirement item TDT 4242 TDT 4242Requirements quality metrics – 4 Requirements quality metrics – 5 Opacity: Requirement items for which rational or  dependencies are invisible. Noise: Requirement items that yield no information on  Multiple unrelated concept mapping. A is not related to B problem world features. X refers to a concept undefined in the domain TDT 4242 TDT 4242
  17. 17. Requirements quality metrics – 6 Requirements quality metrics Completeness: The needs of a prescribed  Quality metrics on a requirements set provides useful  system are fully covered by requirement  understanding, tracking and control of requirements  items without any undesirable outcome. improvement process. No requirement item mentions the goal concept Z TDT 4242Where do the goals come from? Summary We get goals from: Goals can be defined at different levels of • Preliminary analysis of the current system. abstraction • Systematically by searching intentional keywords  There are two types of goals: Behavioral or soft goal in documents provided, interview transcripts etc.  E.g. ‘objective’, ‘purpose’, ‘in order to’. • Iterative refinement and abstraction of high‐level  There are several categories of goals, e.g. goals: By asking the how and why question.  • Functional and non-functional Results in a goal refinement tree • Goal refinement provides a natural mechanism • Approaches: KAOS – Goal driven requirements  for structuring complex specifications at acquisition. different levels of concern: • Goal refinement graph TDT 4242 TDT 4242
  18. 18. Requirements There are three levels of requirements: • Informal – e.g. Natural language (NL): free text, no rules apply Institutt for datateknikk og informasjonsvitenskap • Semiformal • Guided Natural Language (GNL): free text but Inah Omoronyia and Tor Stålhane allowable terms are defined by a vocabulare • Boilerplates (BP): structured text and an ontology – vocabulary plus relationships Guided Natural Language between terms and • Formal: e.g. state diagrams or predicate logic Requirement Boilerplates TDT 4242 TDT 4242 TDT 4242 Requirements elicitation Humans and machines – 1Step 1: Step 2: Step 3: Step 4: Given the amount and complexity of RE, we Capture Transfer Requirements and Refine the requirements model Create aRequirements in functions into a semi-formal and derive detailed requirements preliminary need to automate as much as possible.Natural Language requirements model design model based on the requirement model (to be Humans and machines have different strong Req.012: The systemshall enable cabin Function 1 Function 2 Function 1 used and and weak points.temperature regulation refined inbetween 15°C and 30°C … Req 001 Req 002 Req 011 Req 028 Function 1a Req 001.01 SP3) We want to elicit and analyze requirements in a  Req 012 Req 050 …Req.124: Cabin … … Function 1b Req 001.02 …. way that allows both parties to build on their  Req 124 …temperature shall notexceed 35° Function 1c strong sides. Parallel Steps: Apply dictionary with common vocabulary; validate and check Requirements consistency and completeness
  19. 19. Humans and machines - 2 Why BPs and GNL – 1 Machines are   GNL and BPs will reduce variation and thus giving   good at observing quantitative data and being  the machines the opportunity to do what they are  deductive, fast and precise. In addition, they are  best at: to be fast, precise and consistent.  good at doing consistent repetition of several   By combining humans and machines and let both do  actions. what they are best at, we get a better result than we   bad at handling variations in written material and  would get if we left the job of handling requirements  pattern recognition. to just one of them.  Humans are good at handling variations in written  material, being inductive. In addition, they are good  at doing error correction.Why BPs and GNL - 2 GNL and BPs Template based textual = Syntax + Semantics + Meta Model The final goal is to allow the machine to assist the  Keywords: RMM - Refinement developers in analysing requirements for: Guided RSL Boilerplates Reflects requirement, system and domain Analysis - Specialization concepts -Correctness  Consistency -Completeness Requirements expressed on templates -Consistency  Completeness  Uses predefined templates based on concepts, relations and axioms to guide requirements elicitation -Safety analysis  Safety implications  Example:  The <system function> shall provide <system capability> to achieve <goal>  Requirements expressed using a vocabulary guide Uses predefined concepts, relations and axioms to guide requirements elicitation Example: The ACC system shall be able to determine the speed of the ego-vehicle. Ontology: General and SP specific - Requirements classification - System attributes - Domain concepts
  20. 20. What is GNL - 1 What is GNL - 2 Aim: Free text requirement elicitation with the  • Bridge the gap between unconstrained  assistance of prescribed words from a  expression and quality checking when  dictionary. This will give us requirements  representing requirements as free text. Quality  which use all terms in a uniform way, this  measures: reducing misunderstandings  Correctness, consistency, completeness and un‐ No formal constraints ambiguity (reduced variability) Requires minimal expertise. • Provide the basis for semantic processing and  checking of requirements. • Dictionary – Simple taxonomy or more formal  ontologyApproach for GNL – 1 Approach for GNL – 2Ontology = Thesaurus + Inference Rules Required Activity • Thesaurus – Domain concepts: entities,   Knowledge capture: Information embedded in  domain events from domain experts and ontologist terms and events  Implementation: Formal representation of captured  • Inference Rules – Relations, attributes and  knowledge. Language: OWL, Support environment:  axioms Protégé.  Verification: Checking that represented ontology is  • Causality, similarity, reflexivity,  correct using transitiveness, symmetric, disjoint  • Classifiers/reasoners • Domain experts (semantic accuracy) (contradiction) … • Mapping of requirement segments to ontology concepts
  21. 21. Motivation for use of templates - 1 Motivation for use of templates - 1Text has the advantage of unconstrained Template based textual requirements expression. There is, however, a need for specification (boilerplates) will introduce some common limitations when representing requirements but will also reduce the opportunity to • Understanding of concepts used to express introduce ambiguities and inconsistencies. the requirements and relations between them. Boilerplates • Format of presentation • Provides an initial basis for requirementsLack of common understanding makes checking requirement specifications expressed as free text prone to ambiguous representations and • Are easy to understand for stakeholders inconsistencies. compared to more formal representations TDT 4242 TDT 4242What is a boilerplate – 1 What is a boilerplate – 2Boilerplates is a set of structures that can be used to The RE process is as follows: write requirements. They use high-level concept classification and attributes 1. Select a boilerplate or a sequence of boilerplates. The selection is based on the attributes that need to be included and how they are organized – fixed terms. 2. If needed, identify and include mode boilerplates 3. Instantiate all attributes A boilerplate consists of fixed terms and attributes. It may, or may not, contain one or more modes. TDT 4242 TDT 4242
  22. 22. Fixed Terms Boilerplate examples - 1Attributes BP32 The <user> shall be able to <capability> Attributes: • <user> = driver • <capability> = start the ACC system Requirement The driver shall be able to start the ACC system TDT 4242
  23. 23. Boilerplate examples - 2 Boilerplate examples - 3 BP43 While <operational condition> BP2 BP32 The <user> shall be able to <capability> The <system> shall be able to <action> <entity> BP43 is a mode Attributes: •<system> = ACC system Attributes •<action> = determine • <operational condition> = activated •<entity> = the speed of the ego-vehicle • <user> = driver • <capability> = override engine power control of Requirement the ACC system The ACC system shall be able to determine the Requirement speed of the ego-vehicle While activated the driver shall be able to override engine power control of the ACC-system TDT 4242 TDT 4242 Functional requirements example Non functional requirement example – 1Functional requirements from the SafeLoc system  The robot control system shall stop the robot within 10  Non‐functional requirements and soft goals fits into  milliseconds if a gate is opened to the zone where the robot is  the same BPs as functional requirements operating  The robot shall only be allowed to start when all gates are  BP61 closed and the reset button is pushed The <system> shall be able to <action> to <entity>  The robot shall stop if it tries to move into a zone already  occupied by an operator Suitability:  The <system > shall be able to <provide an  appropriate set of functions> to <the user> TDT 4242 TDT 4242
  24. 24. Non functional requirement example – 2 Non functional requirement example – 3Non‐functional requirements and soft goals fits into  the same BPs as functional requirements BP43 While <operational condition>BP2-1 The <system> shall be able to <capability> BP2BP12 …for a sustained period of at least The <system> shall be able to <action> <entity> <number> < unit> While <normal operational condition> theMaturity: <system> shall be able to <tolerate> The <system > shall be able to <operate without  <90% of software faults of category...> failure> for a sustained period of at least <quantity>  <time unit>  TDT 4242Summing upThe use of boiler plates and ontologies will• Enforce a uniform use of terms• Reduce the variability of presentations – requirements that are similar will look similarReduced variation in form and contents simplifies the use of automatic and semi-automatic tools for• Checking requirement quality – e.g completeness and consistency• Creating test cases TDT 4242
  25. 25. What is requirements traceability “Requirements traceability refers to the ability  to describe and follow the life of a  Institutt for datateknikk og requirement, in both a forwards and  informasjonsvitenskap backwards direction, i.e. from its origins,  Inah Omoronyia and Tor Stålhane through its development and specification, to  its subsequent deployment and use, and  Requirements Traceability through periods of on‐going refinement and  iteration in any of these phases.” Gotel and Finkelstein TDT 4242 TDT 4242 TDT 4242Traceability Goals - 1 Traceability Goals – 2 • Project Management • Validation – Status: “When will we finish?” and “what will it cost?” – finding and removing conflicts between requirements – Quality: “How close are we to our requirements?” – completeness of requirements • derived requirements cover higher level requirements • QA manager • each requirement is covered by part of the product – Improve Quality: “What can we do better?” • Verification • Change Management – assure that all requirements are fulfilled – Versioning, documentation of changes (Why? What?  • System Inspection When?) – identify alternatives and compromises – Change Impact analysis • Certification/Audits • Reuse – proof of being compliant to standards – Variants and product families – Requirements can be targeted for reuse TDT 4242 TDT 4242
  26. 26. Habitat of Traceability Links – 1 Habitat of Traceability Links – 2 Pre- vs. Post-Requirements Specification TDT 4242 TDT 4242Challenges of traceability – 1 Challenges of traceability – 2 – Traces have to be identified and recorded among  numerous, heterogeneous entity instances  – A variety of tool support  (document, models, code, . . . ). It is challenging  • based on traceability matrix, hyperlink, tags  to create meaningful relationships in such a  and identifiers. complex context. • still manual with little automation – Traces are in a constant state of flux since they  – Incomplete trace information is a reality due to  may change whenever requirements or other  complex trace acquisition and maintenance. development artefacts change. – Trust is a big issue:  lack of quality attribute • There is no use of the information that 70% of  trace links are accurate without knowing which  of the links forms the 70% TDT 4242 TDT 4242
  27. 27. Challenges of traceability – 3 Challenges of traceability – 4 Different stakeholders usage viewpoint (different questions  Different stakeholders usage viewpoint (different questions  asked by different stakeholders): asked by different stakeholders): • Validation: • QA management:  – Tracability can be used as a pointer to the quality of  – “how close are we to our requirements” and “what can we  requirements: do better” to improve quality. • Completeness, ambiguity, correctness/noise,  • Change management inconsistency, forward referencing, opacity – Tracking down the effect of each change  to each involved  – Ensures that every requirement has been targeted by at  component that might require adaptations to the change,  least one part of the product recertification or just retesting to proof functionality.  • Verification • Reuse: – Checking that constraints are not violated (in most cases  – Pointing out those aspects of a reused component that  this is an extension of validation context) need to be adapted to the new system requirements. • Certification/Audit – Even the requirements themselves can be targeted for  reuse. • Testing, maintenance (reverse engineering) TDT 4242 TDT 4242Traceability meta-models – 1 Traceability meta-models – 2• A model is an abstraction of phenomena in the real world; a  meta model is yet another abstraction, highlighting properties  of the model itself.• Meta‐models for traceability are often used as the basis for  the traceability methodologies and frameworks: – Define what type of artefacts should be traced. – Define what type of relations could be established between these artefacts.  Traceability Meta Model Low-end traceability TDT 4242 TDT 4242
  28. 28. Traceability meta-models – 3 European EMPRESS project: Meta model for requirements traceabilityHigh-end traceability TDT 4242Traceability meta-models – 4 Approaches to traceability Creating trace links: – Critical tasks in requirements traceability:  establish links between requirements and  between requirements and other artefacts.  – Manual linking and maintaining of such  links is time consuming and error prone – Focus is on requirements traceability  through (semi‐)automatic link generation.  PRECISE Meta‐model (SINTEF)
  29. 29. Manual trace links – 1 Manual trace links – 2This is the classical traceability methods andthe simplest form of traceability. In this approach, we create a requirements traceability matrices using a hypertext or table cross referencing scheme, often using ExcelTwo problems • Long-term difficulty of maintaining a large numbers of links. • The static nature of the links (lack of attributes) limit the scope of potential automation.Scenario driven traceability – 1 Scenario driven traceability – 2 • Test‐based approach to uncover relations amongst  The method to achieve traceablity uses the idea of  requirements, design and code artifacts (Alexander  “footprint”.  Egyed ) When we are dealing with traceability, a footprint   • Accomplished by observing the runtime behavior of  contains two types of information: test scenarios.  • The set of classes that were executed when we were  • IBM Rational PureCoverage, open source tool  testing a specific scenario. (org.jmonde.debug.Trace) • The number of methods that were executed in each  • Translate this behavior into a graph structure to  class.   indicate commonalities among entities  associated with the behavior
  30. 30. Footprints – 1 Footprints – 2E.g. scenario A uses 10 methods in class CAboutDlg Only classes are registered – e.g scenario [s3] uses and 3 methods in Csettings Dlg classes C, J, R and U Footprints – 3 Footprints – 4Some problems: Based on the footprint table, we can make a requirements-to-class trace table• There might be scenarios that do not cover any requirement – e.g. [s3]• There are scenarios that belong to several requirements, e.g. [s9]Such scenarios will get separate rows in the trace matrix and will be marked with an F (Fixed) or a P (Probable), depending on how sure we are that a certain class belongs to this scenario.
  31. 31. Footprints – 5 Development footprints - 1Each test scenario will leave a footprint. If we make A solution that enables the project to construct one test scenario per requirement, then we get one traceability information during development has footprint per requirement. been suggested by I. Omoronyia et al. The method requires that each developerWe can make the footprints more fine grained and • Always identifies which requirement – e.g. use case thus get more information by using methods or – he is currently working on code chunks instead for classes. • Only works at one use case at a timeThis will require more work but also more – better – traceability information. Development footprints - 2 Development footprints - 3The result will be similar to the scenario testing We can extract more info from the development footprint table. process in order to understand better what hasThe resulting table will show which documents, been going on in the project. The next slides shows classes etc. have been accessed during work on • Types of access: C – Create, U – Update and V – this particular requirement – e.g. use case. ViewMain problem: “false” accesses – e.g. a developer • Timeline – e.g. date or time looks at some of the code of another requirement • Person – who did what and, more important, who for info. will have expertise on what? Each line in the table will show
  32. 32. Development footprints - 4 Scenario driven traceability – 3 Problems: – Semi‐automated but requires a large amount of time  from system engineers to iteratively identify a subset  of test scenarios and how they related to  requirement artifacts. – Requirements that are not related due to non  matching execution paths might be related in some  other form (e.g calling, data dependency,  implementation pattern similarity, etc). Trace by tagging – 1 Trace by tagging – 2This method is easy to understand and simple to implement. The problem is that it depends on heavy human intervention.The principle is as follows:• Each requirement is given a tag, either manually or by the tool.• Each document, code chunk, etc. are marked with a tag which tells which requirement it belongs to
  33. 33. Trace by tagging – 3 Trace by tagging – 4There are several ways to create tags, e.g.: The quality of traceability through tagging will depend• Single level tags – e.g. R4. This gives a standard on that we remember to tag all relevant documents. trace matrix It is possible to check automatically that all• Multilevel tags – e.g. R4, R4.1 and R4.2 where R4 documents in the project database is tagged. is the top level requirement and R4.1 and R4.2 are It is, however, not possible to check that this tagging sub-requirements. This gives us more detailed is correct. trace information Conclusion• Requirements traceability is an important aspect of requirements management• Stakeholders have different traceability information needs• Traceability can be complex for not trivial projects• Traceability meta-models provide insight on the type of traceability information required for a project• There exist several automated approaches for requirements traceability. The strength is in a synergy of different automated approaches
  34. 34. Tor StålhaneRequirements Specification and Testing Requirements testability
  35. 35. Testability definitionAccording to ISO 9126, testability is defined as:Testability: The capability of the software product to enable modified software to be validated.NOTE - Values of this sub-characteristic may be altered by the modifications under consideration.
  36. 36. Testability concernsTestability touches upon two areas of concern:• How easy is it to test the implementation?• How test‐friendly is the requirement?These two concerns are not independent and  need to be considered together. We will first  look at it from the requirements side.
  37. 37. TestabilityThree basic ways to check that we have achieved our goals:• Executing a test. Give input, observe and check output. A test can be a – Black box test – White box test – Grey box test• Run experiments• Inspect the code and other artifactsUsually, we will include all of these activities in the term testing
  38. 38. When to use whatThe diagram on the next slide is a high level overview of when to use• T – Tests. Input / output. Involves the computer system and peripherals.• E – Experiments. Input / output but involves also the users.• I – Inspections. Evaluation based on documents.
  39. 39. Concrete requirements from high level goals T EE T I I T E E E TDT 4242 
  40. 40. TestabilityIn order to be testable, a requirement needs to be stated in a precise way. For some requirements this is in place right from the start: When the ACC system is turned on, the “Active” light on the dashboard shall be turned on.In other cases we need to change a requirement to get a testable version. The system shall be easy to use.
  41. 41. Testability challenges - 1Some requirements are more difficult to test than others. Problems might rise due to:• Volume of tests needed, e.g. response time or storage capacity.• Type of event to be tested, e.g. error handling or safety mechanisms.• The required state of the system before testing, e.g. a rare failure state or a certain transaction history.
  42. 42. Testability challenges – 2We can test the requirements at any level. Theformulation of the test will depend on the level
  43. 43. Making a requirement testable – 1One way to make requirements testable is the “Design by Objective” method introduced by Tom Gilb.The method is simple in principle but is in some cases difficult to use. There are two problems• The resulting tests can in some be rather extensive and thus quite costly.• The method requires access to the system’s end users.
  44. 44. Making a requirement testable – 21. What do you mean by <requirement>? This will give us either (a) a testable requirement or (b) a set of testable and non-testable sub-requirements.2. In case (a) we are finished. In case (b) we will repeat question 1 for each non- testable sub-requirement
  45. 45. Making a requirement testable – 3Requirement: Reverse thrust may only be used, when the airplane is landed.The important questions are• “How do you define landed?”• Who should you ask – e.g. pilots, airplane construction engineers, or airplane designers?
  46. 46. Requirements for testability – 1  First and foremost:The customer needs to know what he wants and why he wants it. In some cases it is easier to test if the user actually has achieved his goal than to test that the system implements the requirement.Unfortunately, the “why”-part is usually not stated as part of a requirement.
  47. 47. Requirements for testability – 2Each requirement needs to be• Correct, i.e. without errors• Complete, i.e. has all possible situations been covered?• Consistent, i.e. not in disagreement with other requirements.• Clear, i.e. stated in a way that is easy to read and understand – e.g. using a commonly known notation.
  48. 48. Requirements for testability – 3Each requirement needs to be• Relevant, i.e. pertinent to the system’s purpose and at the right level of restrictiveness.• Feasible, i.e. possible to realize. If it is difficult to implement, is might also be difficult to test.• Traceable, i.e. it must be possible to relate it to one or more – Software components – Process steps
  49. 49. CompletenessAll possible situations must be covered.“If X then….”, “If Y then….” Must also consider what will happen “If neither X nor Y…”Automatic door opener – what is missing?If the door is closed and a person is detected then send signal Open_Door. If no person is detected after 10 sec., send signal Close_Door.
  50. 50. ConsistencyConsistency is a challenge since we, at least in the general case, need a complete overview of all requirements.In most cases, we can make do with checking all requirements that are related to the same event, function or parameter.
  51. 51. Clear – 1This is mainly a question of representation such as choice of• Diagram notation• Description language• Level of detailsWho shall understand the requirement?• Customers• Developers, including hired-in consultants• Testers
  52. 52. Clear – 2Simple example:Print the “accounts ledger” for all accountsThis requirement is perfectly clear for developers who are working in the banking business.Other developers might experience some problems.
  53. 53. RelevantTwo questions are important:• Do we really need this requirement?• Is it at the right level of strictness – i.e. not too strict and not too lax.Only the second question is important for the tester.• Too strict means more work for the developers• Too lax means more work for the tester.
  54. 54. FeasibleThis question is really related to the contract but we should also consider it here – can we really do this?Testers can contribute to the feasibility question by asking how it should be tested. This will help / force everybody involved to make the requirement more clear and thus improve on the requirement.Requirements that are difficult to tests are also usually difficult to implement – mainly because they are badly defined.
  55. 55. Some sound advice The following set of advices on requirements and  testability are quoted from Ludwig Consulting  Services, LLC.They are not a definition and not “the final words” on requirements testability. Instead, they should  be used as a checklist. That one of the following rules are not obeyed does  not mean that the requirement is wrong. It  should, however, be reviewed for potential  problems. 
  56. 56. Modifying PhrasesWords and phrases that include: • as appropriate• if practical• as required• to the extent necessary / practical. Their meaning • is subject to interpretation • make the requirement optional Phrases like "at a minimum" only ensure the minimum,  while "shall be considered" only requires the  contractor to think about it.
  57. 57. Vague WordsVague words inject confusion. Examples of frequently  used vague verbs are: • manage • track• handle • flag Information systems receive, store, calculate, report, and  transmit data. Use words that express what the system  must do. Requirement: The system shall process ABC data to the  extent necessary to store it in an appropriate form for  future access. Correction: The system shall edit ABC data. 
  58. 58. Pronouns With No ReferenceExample: It shall be displayed. When this occurs, the writer is usually relying on  a nearby requirement in the requirements  document for the meaning of "it." As requirements are assigned for  implementation, they are often reordered and  regrouped, and the defining requirement is no  longer nearby.
  59. 59. Passive VoiceRequirements should be written in active voice,  which clearly shows X does or provides Y.Passive voice: Z shall be calculated. Active voice: the system shall calculate Z.
  60. 60. Negative RequirementsEverything outside the system is what the system  does not do. Testing would have to continue forever to prove  that the system does not do something. State what the system does. Substitute an active  verb that expresses what the system must do. • Change "the system shall not allow X," to "the  system shall prevent Y." • Use the prefix "un," such as: The system shall  reject unauthorized users. 
  61. 61. Assumptions and Comparisons – 1 The requirement "the system shall increase  throughput by 15%" sounds testable, but isnt. The assumption is "over current system  throughput." By comparing to another  system, the meaning of the requirement  changes when the other system changes
  62. 62. Assumptions and Comparisons – 2 An example, sometimes found in requests for  proposals, is: "The system shall address the future needs of  users." The writer is probably thinking ahead to after the  contract is awarded. The requirement is  meaningless because whenever it is read, it will  point to the future. A requirement on change management included in  the project management processes, would make  more sense than making it a requirement for the  system.
  63. 63. Indefinite PronounsIndefinite pronouns are “stand in” for unnamed people orthings, which makes their meaning subject to interpretation.Some of these may find their way into requirements:• All • Everybody• Another • Everyone• Any • Everything• Anybody • Few• Anything • Many • Each • Most• Either • Much • Every 
  64. 64. The implementationWe will shortly look at four concerns related to  implementation and testability:• Autonomy of the system under test• Observability of the testing progress• Re‐test efficiency • Test restartability  
  65. 65. Autonomy – 1  How many other systems are needed to test this  requirement? It is best if it can be tested using only the SUT  and the autonomy and testability is  successively reduced as the number of other,  necessary systems increase. If the other systems needed are difficult to  include at the present we will have to write  more or less complex stubs. 
  66. 66. Example – 1  Requirement: “If the door is closed and a person is detected then send signal Open_Door”• Sensors and actuators can be tested in the lab.• The system with a simulated actuator: simulate a “person detected” signal on the sensor and check if a Open_Door signal is sent to the actuator.
  67. 67. Example – 2 We can build a complete system – door, sensor,  door motor and software system and test by• Letting persons approach the sensor• Check if the door opens – Early enough – Fast enough 
  68. 68. Observability How easy is it to observe the • Progress of the test execution? This is important for tests that do not produce  output – e.g. the requirement is only  concerned with an internal state change or  update of a database. • Results of the test? Important for tests where the output is  dependent on an internal state or database  content. 
  69. 69. Re‐test effiencey Retest efficiency is concerned with how easy it is  to perform the “test – check – change – re‐ test” cycle.  This includes • Observability – observe the test result• Traceability – identify all tests related to the  change  
  70. 70. Test restartability This is mostly a question of check points in the  code. How easy is it to • Stop the test temporarily• Study current state and output• Start the test again from – The point where it was stopped – Start 
  71. 71. Final commentsThat a requirement is testable does not necessarily mean that it is easy to test.In order to have testable requirements it is important that• The testers are involved right from the start of the project. It is difficult to add testability later.• The tests are an integrated part of the requirement
  72. 72. Introduction to exercise 1 Tor Stålhane
  73. 73. The goals ‐ 1Consider the following goals for an adaptive  cruse controller – ACC:When the ACC is active, vehicle speed shall be  controlled automatically to maintain one of  the following:• time gap to the forward vehicle • the set speed
  74. 74. The goals ‐ 2In case of several preceding vehicles, the  preceding vehicle will be selected  automaticallyThe ACC shall support a range of types• Type 1a: manual clutch operation• Type 1b: no manual  clutch operation, no  active brake control• Type 2a: manual clutch operation, active  brake control• Type 2b: active brake control
  75. 75. Exercise requirements • Identify the combinations of sub‐goals that  will jointly contribute to each of the goals• Build a refinement graph showing the low‐ level goals, requirements , assumptions,  domain properties and associated agents
  76. 76. ACC state model
  77. 77. Test vs. inspection Part 1  Tor Stålhane
  78. 78. What we will cover• Part 1  – Introduction  – Inspection processes – Testing processes • Part 2 – Tests and inspections – some data – Inspection as a social process – two experiments  and some conclusions
  79. 79. Introduction 
  80. 80. Adam’s data ‐ 1 Mean Time to Problem Occurrence – yearsProduct 1.6 5 16 50 160 500 1600 5000 1 0.7 1.2 2.1 5.0 10.3 17.8 28.8 34.2 2 0.7 1.5 3.2 4.3 9.7 18.2 28.0 34.3 3 0.4 1.4 2.8 6.5 8.7 18.0 28.5 33.7 4 0.1 0.3 2.0 4.4 11.9 18.7 28.5 34.2 5 0.7 1.4 2.9 4.4 9.4 18.4 28.5 34.2 6 0.3 0.8 2.1 5.0 11.5 20.1 28.2 32.0 7 0.6 1.4 2.7 4.5 9.9 18.5 28.5 34.0 8 1.1 1.4 2.7 6.5 11.1 18.4 27.1 31.9 9 0.0 0.5 1.9 5.6 12.8 20.4 27.6 31.2
  81. 81. Adams’ data – 2  The main information that you get from the  table on the previous slide is that• Some defects are important because they will  happen quite often.• Most defects are not important since they will  happen seldom. How can we tell the difference?
  82. 82. Testing and inspection – the V model
  83. 83. Testing and inspection – 1 The important message here is that testing  cannot always be done. In the first, important phases, we have nothing  to execute and will thus always have to do  some type of inspection. This might be considered one of the weaknesses  of traditional software engineering over Agile  development. 
  84. 84. Testing and inspection – 2  In order to understand the main differences  between testing and inspection, we should  consider Fit’s list.Based on this, we will give a short discussion of  the relative merits of testing and inspection.
  85. 85. Area of Man MachinecompetenceUnderstanding Good at handling variations in Bad at handling variations in written material written materialObserve General observations, Specialized, good at observing multifunctional quantitative data, bad at pattern recognitionReasoning Inductive, slow, imprecise but Deductive, fast, precise but good at error correction bad error correctionMemory Innovative, several access Copying, formal access mechanismsInformation Single channel, less than 10 Multi channel, severalhandling bits per second Megabits per secondConsistency Unreliable, get tired, depends Consistent repetition of several on learning actionsPower Low level, maximum ca. 150 High level over long periods watt of timeSpeed Slow – seconds Fast
  86. 86. Man vs. machine – 1  Good when we need the ability to • Handle variation• Be innovative and inductive• Recognize and handle patternsNot so good when we need the ability to• Do the same things over and over again in a  consistent manner• Handle large amount of data
  87. 87. Man vs. machine – 2  In order to do the best job possible we need  processes where we let each part• Do what they are best at: – Man is innovative – Machine handles large amounts of data • Support the other with their specialties. – Machine supports man by making large amounts  of information available – Man support machine by providing it with  innovative input
  88. 88. General considerations ‐ documentsArchitecture, system, sub‐system and  component design plus pseudo code. Here we  can only use inspections. Man will use experience and knowledge to  identify possible problems Machine can support by identifying information  – e.g. find all occurrences of a string. 
  89. 89. General considerations – code (1)For executable code, we can use inspection,  testing or a combination of both. The size and complexity – degree of dynamism – of the code will, to a large degree, decide our  choice.Other important factors are the degree of  experience with • The programming language• The algorithms used 
  90. 90. General considerations – code (2)Simple code• Start with inspection – all code• Design and run testsComplex code• Start with inspection – focus on algorithm and  logic• Decide test completeness criteria – we cannot  test everything• Design and run tests 
  91. 91. Inspection processes
  92. 92. Inspections – 1  The term “inspection” is often used in a rather  imprecise manner. We will look at three types  of inspection:• Walkthrough• Informal inspection – also called informal  review• Formal inspection – also called formal review  or just inspectionThe first two types are usually project internal  while the last one is used as a final acceptance  activity for a document. 
  93. 93. Inspections – 2For all types of inspections:• The quality of the results depends on the  experience and knowledge of the participants.  “Garbage in – Garbage out”• It might be a good idea to involve customer  representatives.
  94. 94. The walkthrough process Walkthrough is a simple process – mostly used  for early decisions for an activity. The  document owner:1. Makes a rough sketch of the solution – architecture, algorithm etc. 2. Presents – explain – the  sketch to whoever  shows up. 3. Registers feedback – improvements.  
  95. 95. Walkthrough – pros and consPros:• Easy and inexpensive. Needs no extra  preparation.• Collect ideas at an early stage of  development.Cons:• No commitment from the participants• May collect many loose or irrelevant ideas 
  96. 96. The informal inspection process Planning Rules, checklists, proceduresProduct  Change Individual Loggingdocument requests checking meeting
  97. 97. Informal inspections – pros and consPros:• Is simple and inexpensive to perform.• Can be used at all stages of development• Usually has a good cost / benefit ratio• Needs a minimum of planningCons:• No participant commitment • No process improvement
  98. 98. The formal inspection processThe formal inspection process described below  is – with small variations – the most  commonly used. The version shown on the  following slides stem from T. Gilb and D.  Graham. We recommend this process as the final  acceptance process for all important  documents
  99. 99. Formal inspection process overview Change requestsProduct document Planning Process  improvements Rules, checklists, procedures Edit  Individual and  Walk‐ Kick‐off Logging follow‐ through checking meeting up
  100. 100. Distribution of resources  TypicalActivity Range % value %Planning 3–5 4Kick-off 4–7 6Individual checking 20 – 30 25Logging 20 – 30 25Editing 15 – 30 20Process brainstorming 15 – 30 16Leader overhead, follow up, entry, 3–5 4exit
  101. 101. Initiating the inspection process• The inspection process starts with a “request  for inspection” from the author to the QA  responsible.• The QA responsible appoints an inspection  leader.• First step is always to check that the  document is fit for inspection.
  102. 102. PlanningImportant planning points are:• Who should participate in the inspections – Who is interested? – Who have time available for preparation and  meetings? – Who has the necessary knowledge concerning  application, language, tools, methods?
  103. 103. Kick‐offImportant activities here are:• Distribution of necessary documents: – Documents that shall be inspected – Requirements – Applicable standards and checklists• Assignment of roles and jobs• Setting targets for resources, deadlines etc.
  104. 104. Individual checkingThis is the main activity of the inspection. Each  participant read the document to look for  • Potential errors ‐ inconsistencies with  requirements or common application  experience• Lack of adherence to company standards or  good workmanship
  105. 105. Logging meetingThe logging meeting has three purposes:• Log issues already discovered by inspection  participants• Discover new issues based on discussions  and new information that arises during the  logging meeting.• Identify possible improvement to the  inspection or development process.
  106. 106. Improve the product ‐ 1The author receives the log from the inspection  meeting. All items ‐ issues ‐ in the log are  categorised as one of the following:• Errors in the author’s document.• Errors in someone else’s document.• Misunderstandings in the inspection team.
  107. 107. Improve the product ‐ 2• Errors in own document: Make appropriate corrections• Errors in someone else’s documents: Inform the owner of this document.• Misunderstandings in the inspection team: Improve document to avoid further  misunderstandings.
  108. 108. Checking the changesThis is the responsibility of the inspection  leader. He must assure that all issues raised  in the log are disposed of in a satisfactory  manner:• The documents that have been inspected• Related documents ‐ including standards  and checklists• Suggested process improvements 
  109. 109. Formal inspection – pros and consPros:• Can be used to formally accept documents• Includes process improvement Cons:• Is time consuming and expensive• Needs extensive planning in order to succeed
  110. 110. Testing processes 
  111. 111. Testing We will look at three types of testing:• Unit testing – does the code behave as  intended. Usually done by the developer• Function verification testing – also called  systems test. Does the component or system  provide the required functionality?• System verification testing – also called  acceptance test. Does the hardware and  software work together to give the user the  intended functionality?
  112. 112. The unit testing processUnit testing is done by the developer one or  more times during development. It is a rather  informal process which mostly run as follows:1.Implement (part of) a component.2.Define one or more tests to activate the code3.Check the results against expectations and  current understanding of the component
  113. 113. Unit testing – pros and consPros:• Simple way to check that the code works.• Can be used together with coding in an  iterative manner.Cons:• Will only test the developer’s understanding  of the spec.• May need stubs or drivers in order to test
  114. 114. The system test processA systems test has the following steps:1. Based on the requirements, identify – Test for each requirement, including error handling – Initial state, expected result and final state2. Identify dependencies between tests3. Identify acceptance criteria for test suite 4. Run tests and check results against  – Acceptance criteria for each test – Acceptance criteria for the test suite
  115. 115. Systems test – pros and consPros:• Tests system’s behavior against customer  requirements.Cons:• It is a black box test. If we find an error, the  systems test must be followed by extensive  debugging
  116. 116. The acceptance test processThe acceptance test usually has three actvities – both involving the customer or his  representatives:• Rerun the systems test at the customer’s site.• Use the system to solve a set of real‐world  tasks.• Try to break the system – by stressing it or by  feeding it large amounts of illegal input
  117. 117. Acceptance test – pros and consPros:• Creates confidence that the system will be  useful for the customer• Shows the system’s ability to operate in the  customer’s environmentCons:• Might force the system to handle input that it  was not designed for, thus creating an  unfavorable impression.
  118. 118. Test vs. inspection Part 2 Tor Stålhane
  119. 119. Testing and inspection A short data analysis 
  120. 120. Test and inspections – some termsFirst we need to understand two important  terms – defect types and triggers.After this we will look at inspection data and  test data from three activity types, organized  according to type of defect and trigger. We need the defect categories to compare test  and inspections – where is what best?
  121. 121. Defect categoriesThis presentation uses eight defect categories:• Wrong or missing assignment• Wrong or missing data validation• Error in algorithm – no design change is necessary• Wrong timing or sequencing• Interface problems• Functional error – design change is needed• Build, package or merge problem• Documentation problem
  122. 122. Triggers We will use different triggers for test and  inspections. In addition – white box and black  box tests will use different triggers. We will get back to triggers and black box /  white box testing later in the course.
  123. 123. Inspection triggers• Design conformance• Understanding details – Operation and semantics – Side effects – Concurrency• Backward compatibility – earlier versions of this  system• Lateral compatibility – other, similar systems• Rare situations• Document consistency and completeness• Language dependencies
  124. 124. Test triggers – black box • Test coverage• Sequencing – two code chunks in sequence• Interaction – two code chunks in parallel• Data variation – variations over a simple test  case• Side effects – unanticipated effects of a simple  test case
  125. 125. Test triggers – white box• Simple path coverage• Combinational path coverage – same path  covered several times but with different  inputs• Side effect ‐ unanticipated effects of a simple  path coverage
  126. 126. Testing and inspection – the V model
  127. 127. Inspection dataWe will look at inspection data from three  development activities:• High level design: architectural design • Low level design: design of subsystems,  components – modules – and data models• Implementation: realization, writing code  This is the left hand side of the V‐model
  128. 128. Test data We will look at test data from three  development activities:• Unit testing: testing a small unit like a method  or a class• Function verification testing: functional  testing of a component, a system or a  subsystem• System verification testing: testing the total  system, including hardware and users.  This is the right hand side of the V‐model
  129. 129. What did we find The next tables will, for each of the assigned  development activities, show the following  information:• Development activity• The three most efficient triggersFirst for inspection and then for testing
  130. 130. Inspection – defect typesActivity Defect type Percentage Documentation 45.10High level design Function 24.71 Interface 14.12 Algorithm 20.72Low level design Function 21.17 Documentation 20.27 Algorithm 21.62Code inspection Documentation 17.42 Function 15.92
  131. 131. Inspection – triggers Activity Trigger Percentage Understand details 34.51High level design Document consistency 20.78 Backward compatible 19.61 Side effects 29.73Low level design Operation semantics 28.38 Backward compatible 12.16 Operation semantics 55.86Code inspection Document consistency 12.01 Design conformance 11.41
  132. 132. Testing – triggers and defects Activity Trigger Percentage Test sequencing 41.90Implementation Test coverage 33.20testing Side effects 11.07Activity Defect type Percentage Interface 39.13Implementation Assignments 17.79testing Build / Package / 14.62 Merge
  133. 133. Some observations – 1  • Pareto’s rule will apply in most cases – both  for defect types and triggers• Defects related to documentation and  functions taken together are the most  commonly found defect types in inspection – HLD: 69.81% – LLD: 41.44% – Code: 33.34%
  134. 134. Some observations – 2 • The only defect type that is among the top  three both for testing and inspection is  “Interface” – Inspection ‐ HLD: 14.12% – Testing: 39.13% • The only trigger that is among the top three  both for testing and inspection is “Side effects” – Inspection – LLD: 29.73 – Testing: 11.07
  135. 135. Summary Testing and inspection are different activities.  By and large, they• Need different triggers• Use different mind sets• Find different types of defects  Thus, we need both activities in order to get a  high quality product
  136. 136. Inspection as a social process 
  137. 137. Inspection as a social processInspections is a people‐intensive process. Thus,  we cannot consider only technical details – we  also need to consider how people• Interact• Cooperate
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×