The document discusses a proposed method for improving the classification of requirements defects by Isabel Margarido, focusing on a scheme to identify frequent and impactful defects, analyze root causes, and reduce risks associated with poor communication and incomplete requirements. It outlines the motivation behind a classification system based on existing practices and quality requirements, suggesting specific practices for requirements development to ensure adherence to standards. The study additionally details various types of defects, emphasizing the necessity for clear definitions to facilitate effective communication and documentation.

1. Portugal
A Method to Improve the Classification
of Requirements Defects
Isabel Margarido (isabel.margarido@gmail.com)
Ph.D. Student Researcher
Faculty of Engineering, University of Porto
João Pascoal Faria
FEUP/INESC
06-07-2012, Coimbra

2. agenda
introduction
Isabel Lopes Margarido, 6th of June 2012, Coimbra
literature review
proposal
assessment
conclusion
2
2/27

3. Isabel Lopes Margarido, 6th of June 2012, Coimbra introduction review proposal assessment conclusion
Explained Understood Designed Coded Sold
Documented Installed Billed Supported Needed
3
3/27

4. Isabel Lopes Margarido, 6th of June 2012, Coimbra introduction review proposal assessment conclusion
v
4
4/27

5. Isabel Lopes Margarido, 6th of June 2012, Coimbra introduction review proposal assessment conclusion
v
5
5/27

6. Isabel Lopes Margarido, 6th of June 2012, Coimbra introduction review proposal assessment conclusion
v
6
6/27

7. Isabel Lopes Margarido, 6th of June 2012, Coimbra introduction review proposal assessment conclusion
7
7/27

8. introduction review proposal assessment conclusion
related CMMI practices


Isabel Lopes Margarido, 6th of June 2012, Coimbra







8
8/27

9. introduction review proposal assessment conclusion
related CMMI practices
maturity level 3 (engineering process areas)


Isabel Lopes Margarido, 6th of June 2012, Coimbra







9
9/27

10. introduction review proposal assessment conclusion
related CMMI practices
maturity level 3 (engineering process areas)
Requirements Development (RD)

Isabel Lopes Margarido, 6th of June 2012, Coimbra







10
10/27

11. introduction review proposal assessment conclusion
related CMMI practices
maturity level 3 (engineering process areas)
Requirements Development (RD)
 Specific Practice (SP) 3.3 “Analyse Requirements”
Isabel Lopes Margarido, 6th of June 2012, Coimbra







11
11/27

12. introduction review proposal assessment conclusion
related CMMI practices
maturity level 3 (engineering process areas)
Requirements Development (RD)
 Specific Practice (SP) 3.3 “Analyse Requirements”
Isabel Lopes Margarido, 6th of June 2012, Coimbra
ensure that they are necessary and sufficient






12
12/27

13. introduction review proposal assessment conclusion
related CMMI practices
maturity level 3 (engineering process areas)
Requirements Development (RD)
 Specific Practice (SP) 3.3 “Analyse Requirements”
Isabel Lopes Margarido, 6th of June 2012, Coimbra
ensure that they are necessary and sufficient
Verification (VER)





13
13/27

14. introduction review proposal assessment conclusion
related CMMI practices
maturity level 3 (engineering process areas)
Requirements Development (RD)
 Specific Practice (SP) 3.3 “Analyse Requirements”
Isabel Lopes Margarido, 6th of June 2012, Coimbra
ensure that they are necessary and sufficient
Verification (VER)
 SP 1.3 “Establish Verification Procedures and Criteria”




14
14/27

15. introduction review proposal assessment conclusion
related CMMI practices
maturity level 3 (engineering process areas)
Requirements Development (RD)
 Specific Practice (SP) 3.3 “Analyse Requirements”
Isabel Lopes Margarido, 6th of June 2012, Coimbra
ensure that they are necessary and sufficient
Verification (VER)
 SP 1.3 “Establish Verification Procedures and Criteria”
 SP 2.1 “Prepare for Peer Reviews”



15
15/27

16. introduction review proposal assessment conclusion
related CMMI practices
maturity level 3 (engineering process areas)
Requirements Development (RD)
 Specific Practice (SP) 3.3 “Analyse Requirements”
Isabel Lopes Margarido, 6th of June 2012, Coimbra
ensure that they are necessary and sufficient
Verification (VER)
 SP 1.3 “Establish Verification Procedures and Criteria”
 SP 2.1 “Prepare for Peer Reviews”
 SP 2.2 “Conduct Peer Reviews”


16
16/27

17. introduction review proposal assessment conclusion
related CMMI practices
maturity level 3 (engineering process areas)
Requirements Development (RD)
 Specific Practice (SP) 3.3 “Analyse Requirements”
Isabel Lopes Margarido, 6th of June 2012, Coimbra
ensure that they are necessary and sufficient
Verification (VER)
 SP 1.3 “Establish Verification Procedures and Criteria”
 SP 2.1 “Prepare for Peer Reviews”
 SP 2.2 “Conduct Peer Reviews”
 SP 2.3 “Analyse Peer Review Data”

17
17/27

18. introduction review proposal assessment conclusion
related CMMI practices
maturity level 3 (engineering process areas)
Requirements Development (RD)
 Specific Practice (SP) 3.3 “Analyse Requirements”
Isabel Lopes Margarido, 6th of June 2012, Coimbra
ensure that they are necessary and sufficient
Verification (VER)
 SP 1.3 “Establish Verification Procedures and Criteria”
 SP 2.1 “Prepare for Peer Reviews”
 SP 2.2 “Conduct Peer Reviews”
 SP 2.3 “Analyse Peer Review Data”
maturity level 5

18
18/27

19. introduction review proposal assessment conclusion
related CMMI practices
maturity level 3 (engineering process areas)
Requirements Development (RD)
 Specific Practice (SP) 3.3 “Analyse Requirements”
Isabel Lopes Margarido, 6th of June 2012, Coimbra
ensure that they are necessary and sufficient
Verification (VER)
 SP 1.3 “Establish Verification Procedures and Criteria”
 SP 2.1 “Prepare for Peer Reviews”
 SP 2.2 “Conduct Peer Reviews”
 SP 2.3 “Analyse Peer Review Data”
maturity level 5
Causal Analysis and Resolution (CAR)
19
19/27

20. introduction review proposal assessment conclusion
motivation
Higher-severity Problem Factors
Isabel Lopes Margarido, 6th of June 2012, Coimbra
(Chen and Huang, 2009)
20
20/27

21. introduction review proposal assessment conclusion
motivation
Higher-severity Problem Factors
Isabel Lopes Margarido, 6th of June 2012, Coimbra
(Chen and Huang, 2009)
(Hamill and Goseva-Popstojanova, 2009)
21
21/27

22. introduction review proposal assessment conclusion
motivation
“classifying or grouping problems helps to indentify clusters in which
systematic errors are likely to be found” (Card, 1998)
 our goal is to define classification scheme for requirements defects
Isabel Lopes Margarido, 6th of June 2012, Coimbra
that facilitates
 identification of more frequent defects with higher impact
 analysis of root causes
 preparation of reviews checklists
 reduction of risks (bad communication, incomplete requirements,
final acceptance difficulties)

22
22/27

23. introduction review proposal assessment conclusion
motivation
“classifying or grouping problems helps to indentify clusters in which
systematic errors are likely to be found” (Card, 1998)
 our goal is to define classification scheme for requirements defects
Isabel Lopes Margarido, 6th of June 2012, Coimbra
that facilitates
 identification of more frequent defects with higher impact
 analysis of root causes
 preparation of reviews checklists
 reduction of risks (bad communication, incomplete requirements,
final acceptance difficulties)
ODC
(Chillarege et al., 1992)

HP
(Grady, 1976)
23
23/27

24. introduction review proposal assessment conclusion
motivation
“classifying or grouping problems helps to indentify clusters in which
systematic errors are likely to be found” (Card, 1998)
 our goal is to define classification scheme for requirements defects
Isabel Lopes Margarido, 6th of June 2012, Coimbra
that facilitates
 identification of more frequent defects with higher impact
 analysis of root causes
 preparation of reviews checklists
 reduction of risks (bad communication, incomplete requirements,
final acceptance difficulties)
ODC
(Chillarege et al., 1992)
 more adequate for other phases than the
requirements phase
HP
(Grady, 1976)
24
24/27

25. introduction review proposal assessment conclusion
requirements review
Isabel Lopes Margarido, 6th of June 2012, Coimbra
25
25/27

26. introduction review proposal assessment conclusion
quality requirements for classification schemes
Isabel Lopes Margarido, 6th of June 2012, Coimbra
26
26/27

27. introduction review proposal assessment conclusion
quality requirements for classification schemes
Isabel Lopes Margarido, 6th of June 2012, Coimbra
clearly and meaningfully define attributes
27
27/27

28. introduction review proposal assessment conclusion
quality requirements for classification schemes
complete: every defect is classifiable using the scheme
Isabel Lopes Margarido, 6th of June 2012, Coimbra
28
28/27

29. introduction review proposal assessment conclusion
quality requirements for classification schemes
Isabel Lopes Margarido, 6th of June 2012, Coimbra
attributes values:
• clear and meaningful definition
• small number (5-9)
• aggregate to reduce ambiguity
(Freimut et al., 2005)
• unambiguous
29
29/27

30. introduction review proposal assessment conclusion
phase 1 – assemble classifiers list
 review literature to compile list of existent classifiers and
remove
 the ones that do not apply to the phase or document
Isabel Lopes Margarido, 6th of June 2012, Coimbra
 vague and generic classifiers
 overdetailed
 duplicates (classifiers with same meaning)
 define each classifier and give examples, eliminate ambiguity
through definition
30
30/27

31. introduction review proposal assessment conclusion
type of defect
Isabel Lopes Margarido, 6th of June 2012, Coimbra
31
31/27

32. introduction review proposal assessment conclusion
type of defect
•Not in current baseline
•Out of scope
•Missing/Omission
•Incomplete
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate
•Incorrect
•Inconsistent
•Incompatible
•New (Bell and Thayer, 1976)
•Changed Requirement
•Typos/Clerical
•Unclear
32
32/27

33. introduction review proposal assessment conclusion
type of defect
•Not in current baseline
•Out of scope
•Missing/Omission
•Incomplete
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate
•Incorrect
•Inconsistent
•Incompatible
•New (Bell and Thayer, 1976)
•Changed Requirement
(Basilli and Weiss, 1981)
•Typos/Clerical
•Unclear
•Ambiguity
•Wrong Section/Misplaced
•Other
33
33/27

34. introduction review proposal assessment conclusion
type of defect
•Not in current baseline
•Out of scope
•Missing/Omission
•Incomplete
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate
•Incorrect
•Inconsistent
•Incompatible
•New (Bell and Thayer, 1976)
•Changed Requirement
(Basilli and Weiss, 1981)
•Typos/Clerical
•Unclear (Walia and Craver, 2007)
•Ambiguity
•Wrong Section/Misplaced
•Other
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
•General 34
34/27

35. introduction review proposal assessment conclusion
type of defect
•Not in current baseline •Missing Interface
•Out of scope •Missing Function/Description
•Missing/Omission
•Incomplete
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate
•Incorrect
•Inconsistent
•Incompatible
•New (Bell and Thayer, 1976)
•Changed Requirement
(Basilli and Weiss, 1981)
•Typos/Clerical
•Unclear (Walia and Craver, 2007)
•Ambiguity (Ackerman et al., 1989)
•Wrong Section/Misplaced
•Other
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
•General 35
35/27

36. introduction review proposal assessment conclusion
type of defect
•Not in current baseline •Missing Interface
•Out of scope •Missing Function/Description
•Missing/Omission
•Incomplete •Missing/Incorrect Checking
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate •Missing/Incorrect
•Incorrect Assignment
•Inconsistent •Missing/Incorrect
•Incompatible Timing/Serialization
•New •Missing/Incorrect (Bell and Thayer, 1976)
•Changed Requirement Build/Package/Merge
(Basilli and Weiss, 1981)
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation (Walia and Craver, 2007)
•Ambiguity •Missing/Incorrect Algorithm (Ackerman et al., 1989)
•Wrong Section/Misplaced (Chillarege et al., 1992)
•Other
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
•General 36
36/27

37. introduction review proposal assessment conclusion
type of defect
•Not in current baseline •Missing Interface
•Out of scope •Missing Function/Description
•Missing/Omission
•Incomplete •Missing/Incorrect Checking
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate •Missing/Incorrect
•Incorrect Assignment
•Inconsistent •Missing/Incorrect
•Incompatible Timing/Serialization
•New •Missing/Incorrect (Bell and Thayer, 1976)
•Changed Requirement Build/Package/Merge
(Basilli and Weiss, 1981)
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation (Walia and Craver, 2007)
•Ambiguity •Missing/Incorrect Algorithm (Ackerman et al., 1989)
•Wrong Section/Misplaced •Missing Functionality/Feature (Chillarege et al., 1992)
•Other •Missing Software Interface
(Grady, 1992)
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface
•Redundant/Duplicate •Missing
•General Requirement/Specification 37
37/27

38. introduction review proposal assessment conclusion
type of defect
•Not in current baseline •Incorrect or Extra
•Missing Interface
•Out of scope Functionality
•Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate •Missing/Incorrect
•Incorrect Assignment
•Inconsistent •Missing/Incorrect
•Incompatible Timing/Serialization
•New •Missing/Incorrect (Bell and Thayer, 1976)
•Changed Requirement Build/Package/Merge
(Basilli and Weiss, 1981)
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation (Walia and Craver, 2007)
•Ambiguity •Missing/Incorrect Algorithm (Ackerman et al., 1989)
•Wrong Section/Misplaced •Missing Functionality/Feature (Chillarege et al., 1992)
•Other •Missing Software Interface
(Grady, 1992)
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface (Porter et al., 1992)
•Redundant/Duplicate •Missing
•General Requirement/Specification 38
38/27

39. introduction review proposal assessment conclusion
type of defect
•Not in current baseline •Incorrect or Extra
•Missing Interface
•Out of scope Functionality
•Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
•Not Traceable
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate •Missing/Incorrect
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
•Incompatible Timing/Serialization
•New •Missing/Incorrect (Bell and Thayer, 1976)
•Changed Requirement Build/Package/Merge
(Basilli and Weiss, 1981)
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation (Walia and Craver, 2007)
•Ambiguity •Missing/Incorrect Algorithm (Ackerman et al., 1989)
•Wrong Section/Misplaced •Missing Functionality/Feature (Chillarege et al., 1992)
•Other •Missing Software Interface
(Grady, 1992)
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface (Porter et al., 1992)
•Redundant/Duplicate •Missing (Hayes et al., 2003/6)
•General Requirement/Specification 39
39/27

40. introduction review proposal assessment conclusion
type of defect
•Not in current baseline •Incorrect or Extra
•Missing Interface
•Out of scope Functionality
•Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
•Not Traceable
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate •Missing/Incorrect
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
•Incompatible Timing/Serialization •Extraneous Information
•New •Missing/Incorrect (Bell and Thayer, 1976)
•Changed Requirement Build/Package/Merge
(Basilli and Weiss, 1981)
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation (Walia and Craver, 2007)
•Ambiguity •Missing/Incorrect Algorithm (Ackerman et al., 1989)
•Wrong Section/Misplaced •Missing Functionality/Feature (Chillarege et al., 1992)
•Other •Missing Software Interface
(Grady, 1992)
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface (Porter et al., 1992)
•Redundant/Duplicate •Missing (Hayes et al., 2003/6)
•General Requirement/Specification 40
(Kalinowski et al., 2010)
40/27

41. introduction review proposal assessment conclusion
type of defect
•Not in current baseline •Incorrect or Extra
•Missing Interface
•Out of scope Functionality
•Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate •Missing/Incorrect •Not Traceable
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
•Incompatible Timing/Serialization •Extraneous Information
•New •Missing/Incorrect
•Changed Requirement Build/Package/Merge
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation
•Ambiguity •Missing/Incorrect Algorithm
•Wrong Section/Misplaced •Missing Functionality/Feature
•Other •Missing Software Interface
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface
•Redundant/Duplicate •Missing
•General Requirement/Specification 41
41/27

42. introduction review proposal assessment conclusion
type of defect
•Not in current baseline •Incorrect or Extra
•Missing Interface
•Out of scope Functionality
•Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate •Missing/Incorrect •Not Traceable
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
•Incompatible Timing/Serialization •Extraneous Information
•New •Missing/Incorrect
•Changed Requirement Build/Package/Merge change management
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation
•Ambiguity •Missing/Incorrect Algorithm
•Wrong Section/Misplaced •Missing Functionality/Feature
•Other •Missing Software Interface
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface
•Redundant/Duplicate •Missing
•General Requirement/Specification 42
42/27

43. introduction review proposal assessment conclusion
type of defect
•Incorrect or Extra
•Missing Interface
•Out of scope Functionality
•Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate •Missing/Incorrect
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
•Incompatible Timing/Serialization •Extraneous Information
•Missing/Incorrect
Build/Package/Merge change management
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation
•Ambiguity •Missing/Incorrect Algorithm
•Wrong Section/Misplaced •Missing Functionality/Feature
•Other •Missing Software Interface
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface
•Redundant/Duplicate •Missing
•General Requirement/Specification 43
43/27

44. introduction review proposal assessment conclusion
type of defect
•Incorrect or Extra
•Missing Interface
•Out of scope Functionality
•Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Inadequate •Missing/Incorrect
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
•Incompatible Timing/Serialization •Extraneous Information
•Missing/Incorrect
Build/Package/Merge
•Typos/Clerical •Missing/Incorrect
vague
•Unclear Documentation
•Ambiguity •Missing/Incorrect Algorithm
•Wrong Section/Misplaced •Missing Functionality/Feature
•Other •Missing Software Interface
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface
•Redundant/Duplicate •Missing
•General Requirement/Specification 44
44/27

45. introduction review proposal assessment conclusion
type of defect
•Incorrect or Extra
•Missing Interface
•Out of scope Functionality
•Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Missing/Incorrect
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
•Incompatible Timing/Serialization •Extraneous Information
•Missing/Incorrect
Build/Package/Merge
•Typos/Clerical •Missing/Incorrect
vague
•Unclear Documentation
•Ambiguity •Missing/Incorrect Algorithm
•Wrong Section/Misplaced •Missing Functionality/Feature
•Missing Software Interface
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface
•Redundant/Duplicate •Missing
Requirement/Specification 45
45/27

46. introduction review proposal assessment conclusion
type of defect
•Incorrect or Extra
•Missing Interface
•Out of scope Functionality
•Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Missing/Incorrect
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
•Incompatible Timing/Serialization •Extraneous Information
•Missing/Incorrect
Build/Package/Merge
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation
subsumed
•Ambiguity •Missing/Incorrect Algorithm
•Wrong Section/Misplaced •Missing Functionality/Feature
•Missing Software Interface
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface
•Redundant/Duplicate •Missing
Requirement/Specification 46
46/27

47. introduction review proposal assessment conclusion
type of defect
•Incorrect or Extra
•Missing Interface
•Out of scope Functionality
•Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Missing/Incorrect
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
Timing/Serialization •Extraneous Information
•Missing/Incorrect
Build/Package/Merge
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation
subsumed
•Ambiguity •Missing/Incorrect Algorithm
•Wrong Section/Misplaced •Missing Functionality/Feature
•Missing Software Interface
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface
•Redundant/Duplicate •Missing
Requirement/Specification 47
47/27

48. introduction review proposal assessment conclusion
type of defect
•Incorrect or Extra
•Missing Interface
•Out of scope Functionality
•Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Missing/Incorrect
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
Timing/Serialization •Extraneous Information
•Missing/Incorrect
Build/Package/Merge
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation
•Ambiguity •Missing/Incorrect Algorithm
generic
•Wrong Section/Misplaced •Missing Functionality/Feature
•Missing Software Interface
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface
•Redundant/Duplicate •Missing
Requirement/Specification 48
48/27

49. introduction review proposal assessment conclusion
type of defect
•Missing Interface
•Out of scope •Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Missing/Incorrect
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
Timing/Serialization •Extraneous Information
•Missing/Incorrect
Build/Package/Merge
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation
•Ambiguity •Missing/Incorrect Algorithm
generic
•Wrong Section/Misplaced •Missing Functionality/Feature
•Missing Software Interface
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface
•Redundant/Duplicate •Missing
Requirement/Specification 49
49/27

50. introduction review proposal assessment conclusion
type of defect
•Missing Interface
•Out of scope •Missing Function/Description
•Missing/Omission •Data Type Consistency
•Incomplete •Missing/Incorrect Checking •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Missing/Incorrect
•Incorrect Assignment •Unachievable
•Inconsistent •Missing/Incorrect •Intentional Deviation
Timing/Serialization •Extraneous Information
•Missing/Incorrect
Build/Package/Merge
•Typos/Clerical •Missing/Incorrect
•Unclear Documentation
•Ambiguity •Missing/Incorrect Algorithm
•Wrong Section/Misplaced •Missing Functionality/Feature
•Missing Software Interface over detailed
•Infeasible •Missing Hardware Interface
•Untestable/Non-verifiable •Missing User Interface
•Redundant/Duplicate •Missing
Requirement/Specification 50
50/27

51. introduction review proposal assessment conclusion
type of defect
•Out of scope
•Missing/Omission
•Incomplete •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Incorrect •Unachievable
•Inconsistent •Intentional Deviation
•Extraneous Information
•Typos/Clerical
•Unclear
•Ambiguity
•Wrong Section/Misplaced
over detailed
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
51
51/27

52. introduction review proposal assessment conclusion
•Out of scope •Over-specification
•Missing/Omission
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Unachievable
•Incomplete •Intentional Deviation
•Incorrect
•Inconsistent •Extraneous Information
•Typos/Clerical
•Unclear
•Ambiguity
•Wrong Section/Misplaced
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
52
52/27

53. introduction review proposal assessment conclusion
•Out of scope
•Missing/Omissio •Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
n •Unachievable
•Intentional Deviation
•Incomplete
•Incorrect •Extraneous Information
•Inconsistent
•Typos/Clerical
•Unclear
•Ambiguity
•Wrong Section/Misplaced
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
53
53/27

54. introduction review proposal assessment conclusion
•Out of scope
•Over-specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Unachievable
•Intentional Deviation
•Incorrect •Extraneous Information
•Inconsistent •Missing or Incomplete
•Typos/Clerical
•Unclear
•Ambiguity
•Wrong Section/Misplaced
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
54
54/27

55. introduction review proposal assessment conclusion
•Out of scope •Over-
specification
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Unachievable
•Intentional
•Extraneous
•Incorrect Deviation
Information
•Inconsistent •Missing or Incomplete
•Typos/Clerical
•Ambiguity
•Unclear
•Wrong Section/Misplaced
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
55
55/27

56. Isabel Lopes Margarido, 6th of June 2012, Coimbra introduction review proposal assessment conclusion
•Unachievable
•Incorrect
•Inconsistent •Missing or Incomplete
•Typos/Clerical •Not Relevant or Extraneous
•Unclear
•Ambiguity
•Wrong Section/Misplaced
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
56
56/27

57. Isabel Lopes Margarido, 6th of June 2012, Coimbra introduction review proposal assessment conclusion
•Unachievable
•Incorrect
•Inconsistent •Missing or Incomplete
•Typos/Clerical •Not Relevant or Extraneous
•Ambiguity
•Unclear
•Wrong Section/Misplaced
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
57
57/27

58. Isabel Lopes Margarido, 6th of June 2012, Coimbra introduction review proposal assessment conclusion
•Unachievable
•Incorrect
•Inconsistent •Missing or Incomplete
•Typos/Clerical •Not Relevant or Extraneous
•Ambiguous or Unclear
•Wrong Section/Misplaced
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
58
58/27

59. Isabel Lopes Margarido, 6th of June 2012, Coimbra introduction review proposal assessment conclusion
•Unachievable
•Incorrect
•Inconsistent •Missing or Incomplete
•Typos/Clerical •Not Relevant or Extraneous
•Ambiguous or Unclear
•Wrong Section/Misplaced
•Infeasible
•Untestable/Non-verifiable
•Redundant/Duplicate
59
59/27

60. Isabel Lopes Margarido, 6th of June 2012, Coimbra introduction review proposal assessment conclusion
•Unachievable
•Incorrect
•Inconsistent •Missing or Incomplete
•Typos/Clerical •Not Relevant or Extraneous
•Ambiguous or Unclear
•Infeasible
•Wrong Section/Misplaced
•Untestable/Non-
verifiable
•Redundant/Duplicate
60
60/27

61. introduction review proposal assessment conclusion
•Unachievable
Isabel Lopes Margarido, 6th of June 2012, Coimbra
•Incorrect
•Inconsistent •Missing or Incomplete
•Typos/Clerical •Not Relevant or Extraneous
•Ambiguous or Unclear
•Infeasible
•Wrong Section/Misplaced
•Untestable/Non-
verifiable
•Redundant/Duplicate
61
61/27

62. Isabel Lopes Margarido, 6th of June 2012, Coimbra introduction review proposal assessment conclusion
•Incorrect
•Inconsistent •Missing or Incomplete
•Typos/Clerical •Not Relevant or Extraneous
•Ambiguous or Unclear
•Wrong Section/Misplaced
•Infeasible or Non-verifiable
•Redundant/Duplicate
62
62/27

63. introduction review proposal assessment conclusion
Missing or Incomplete
Isabel Lopes Margarido, 6th of June 2012, Coimbra
Incorrect Information
Inconsistent
Ambiguous or Unclear
Misplaced
Infeasible or Non-verifiable
Redundant or Duplicate
Typo or Formatting
Not Relevant or Extraneous
63
63/27

64. introduction review proposal assessment conclusion
phase 2 – validate classification list
train pilots
 understand classifiers, distinguish them, can use them in
practice
Isabel Lopes Margarido, 6th of June 2012, Coimbra
 use examples (apply classification scheme)
conduct experiment
 same document, team A trained and using new
classifiers, team B not trained nor using new classifiers
analyse results
 number detected defects
 number similar classifications to same defect description
 classifiers systematically confounded
64
64/27

65. introduction review proposal assessment conclusion
phase 3 – readjust classifiers
 adjust definitions, examples, words
 repeat phase 2
Isabel Lopes Margarido, 6th of June 2012, Coimbra
repeat phase 3 if necessary
65
65/27

66. introduction review proposal assessment conclusion
experiments
group A:
19 MSc students
Isabel Lopes Margarido, 6th of June 2012, Coimbra
group B:
6 undergrad
students
66
66/27

67. introduction review proposal assessment conclusion
results
 H0 - all subjects use the same value to classify the type of a
defect
H1 - not all subjects use the same value to classify the type of
Isabel Lopes Margarido, 6th of June 2012, Coimbra
a defect
67
67/27

68. introduction review proposal assessment conclusion
results
 H0 - all subjects use the same value to classify the type of a
defect
H1 - not all subjects use the same value to classify the type of
Isabel Lopes Margarido, 6th of June 2012, Coimbra
a defect
Fleiss Kappa:
moderate (0,46
and 0,44)
68
68/27

69. introduction review proposal assessment conclusion
results
 H0 - all subjects use the same value to classify the type of a
defect
H1 - not all subjects use the same value to classify the type of
Isabel Lopes Margarido, 6th of June 2012, Coimbra
a defect
 Cochran: 0,60 and
0,63 p-value > 0,05
 H0 cannot be
rejected
Fleiss Kappa:
moderate (0,46
and 0,44)
69
69/27

70. introduction review proposal assessment conclusion
results
 H0 - all subjects use the same value to classify the type of a
defect
H1 - not all subjects use the same value to classify the type of
Isabel Lopes Margarido, 6th of June 2012, Coimbra
a defect
 Cochran: 0,60 and
0,63 p-value > 0,05
 H0 cannot be
rejected
Fleiss Kappa:
moderate (0,46
and 0,44)
McNemar: similar results
70
70/27

71. introduction review proposal assessment conclusion
phase 4 – pilot
in pilot teams use the least in a full development cylce
control
Isabel Lopes Margarido, 6th of June 2012, Coimbra
 number of defects from requirements detected on
subsequent phases
 compare with number of defects before using classifiers
list
make adjustments to the classifiers list as necessary (as in
phase 3) and repeat phase 4
71
71/27

72. introduction review proposal assessment conclusion
phase 5 – deploy
update tools
 include tool tips so people can remember definitions
 include definitions and examples on help
Isabel Lopes Margarido, 6th of June 2012, Coimbra
test tools
 use them in practice to detect bugs and necessary
improvements
train all teams (phase 2)
deploy tools
control
 number of defects from requirements detected on
subsequent phases
72
72/27

73. introduction review proposal assessment conclusion
recommendations
people should be trained in the usage of the defects
classification focusing on
 distinction of classifiers
Isabel Lopes Margarido, 6th of June 2012, Coimbra
 clarification of definitions
 practical examples and exercises
avoid choosing a classifier based on its name only
 definition easily available
 e.g., as a tool tip in a tool
73
73/27

74. introduction review proposal assessment conclusion
summary


Isabel Lopes Margarido, 6th of June 2012, Coimbra









74
74/27

75. introduction review proposal assessment conclusion
summary
proposed and assessed a requirements defects classification
scheme based on a literature review and quality guidelines

Isabel Lopes Margarido, 6th of June 2012, Coimbra









75
75/27

76. introduction review proposal assessment conclusion
summary
proposed and assessed a requirements defects classification
scheme based on a literature review and quality guidelines
defect taxonomy needs to support specific organisation
Isabel Lopes Margarido, 6th of June 2012, Coimbra
needs (Card)









76
76/27

77. introduction review proposal assessment conclusion
summary
proposed and assessed a requirements defects classification
scheme based on a literature review and quality guidelines
defect taxonomy needs to support specific organisation
Isabel Lopes Margarido, 6th of June 2012, Coimbra
needs (Card)
organisations should follow proposed methodology when
deciding on classification schemes








77
77/27

78. introduction review proposal assessment conclusion
summary
proposed and assessed a requirements defects classification
scheme based on a literature review and quality guidelines
defect taxonomy needs to support specific organisation
Isabel Lopes Margarido, 6th of June 2012, Coimbra
needs (Card)
organisations should follow proposed methodology when
deciding on classification schemes
type of defect should follow quality properties and support







78
78/27

79. introduction review proposal assessment conclusion
summary
proposed and assessed a requirements defects classification
scheme based on a literature review and quality guidelines
defect taxonomy needs to support specific organisation
Isabel Lopes Margarido, 6th of June 2012, Coimbra
needs (Card)
organisations should follow proposed methodology when
deciding on classification schemes
type of defect should follow quality properties and support
 CAR






79
79/27

80. introduction review proposal assessment conclusion
summary
proposed and assessed a requirements defects classification
scheme based on a literature review and quality guidelines
defect taxonomy needs to support specific organisation
Isabel Lopes Margarido, 6th of June 2012, Coimbra
needs (Card)
organisations should follow proposed methodology when
deciding on classification schemes
type of defect should follow quality properties and support
 CAR
 creation of checklists





80
80/27

81. introduction review proposal assessment conclusion
summary
proposed and assessed a requirements defects classification
scheme based on a literature review and quality guidelines
defect taxonomy needs to support specific organisation
Isabel Lopes Margarido, 6th of June 2012, Coimbra
needs (Card)
organisations should follow proposed methodology when
deciding on classification schemes
type of defect should follow quality properties and support
 CAR
 creation of checklists
 prevent risks resulting from requirements defects




81
81/27

82. introduction review proposal assessment conclusion
summary
proposed and assessed a requirements defects classification
scheme based on a literature review and quality guidelines
defect taxonomy needs to support specific organisation
Isabel Lopes Margarido, 6th of June 2012, Coimbra
needs (Card)
organisations should follow proposed methodology when
deciding on classification schemes
type of defect should follow quality properties and support
 CAR
 creation of checklists
 prevent risks resulting from requirements defects
different people may classify same defect in a different way



82
82/27

83. introduction review proposal assessment conclusion
summary
proposed and assessed a requirements defects classification
scheme based on a literature review and quality guidelines
defect taxonomy needs to support specific organisation
Isabel Lopes Margarido, 6th of June 2012, Coimbra
needs (Card)
organisations should follow proposed methodology when
deciding on classification schemes
type of defect should follow quality properties and support
 CAR
 creation of checklists
 prevent risks resulting from requirements defects
different people may classify same defect in a different way
problems in defects classification


83
83/27

84. introduction review proposal assessment conclusion
summary
proposed and assessed a requirements defects classification
scheme based on a literature review and quality guidelines
defect taxonomy needs to support specific organisation
Isabel Lopes Margarido, 6th of June 2012, Coimbra
needs (Card)
organisations should follow proposed methodology when
deciding on classification schemes
type of defect should follow quality properties and support
 CAR
 creation of checklists
 prevent risks resulting from requirements defects
different people may classify same defect in a different way
problems in defects classification
 different interpretations

84
84/27

85. introduction review proposal assessment conclusion
summary
proposed and assessed a requirements defects classification
scheme based on a literature review and quality guidelines
defect taxonomy needs to support specific organisation
Isabel Lopes Margarido, 6th of June 2012, Coimbra
needs (Card)
organisations should follow proposed methodology when
deciding on classification schemes
type of defect should follow quality properties and support
 CAR
 creation of checklists
 prevent risks resulting from requirements defects
different people may classify same defect in a different way
problems in defects classification
 different interpretations
 misleading analysis
85
85/27

86. Isabel Lopes Margarido, 6th of June 2012, Coimbra
questions
86
86/27

87. images
http://softwareandme.files.wordpress.com/2009/10/3260585819-
project_management.jpg – 21-04-2010
http://www.screenhog.com/sketch/LightbulbIdea.jpg – 21-04-2010
http://igraduatedwhatnow.files.wordpress.com/2009/11/thank_you_sm
all.jpg – 02-05-2010
Isabel Lopes Margarido, 6th of June 2012, Coimbra
http://cartoontester.blogspot.com/2010/01/big-bugs.html – 01-06-2011
http://4.bp.blogspot.com/_eCShgHga-
_g/TPA5KRac8_I/AAAAAAAAH3o/mgw0g75jOus/s400/disagreement.
jpg – 13-06-2011
http://chaospet.com/comics/2008-06-16-90.png – adapted, 13-06-
2011
http://3.bp.blogspot.com/-
_Z2dYcXxMmA/TbLat4c6i_I/AAAAAAAAAnk/KlLdgG-
dgtw/s1600/whereamigoing.jpg – adapted, 25-05-20111
http://www.veryhappypig.com/blog/results.jpg -06-06-2011
http://www.stampa.unibocconi.it/immagini/LA4_economiaq201006031
45905.jpg - 06-06-2011
87
87/27

88. references
 J.-C. Chen and S.-J. Huang, "An empirical analysis of the impact of
software development problem factors on software maintainability,"
Journal of Systems and Software, vol. 82, pp. 981-992 June 2009.
 M. Hamill and G.-P. Katerina, "Common Trends in Software Fault and
Failure Data," IEEE Trans. Softw. Eng., vol. 35, pp. 484-496, 2009.
 R. Chillarege, et al., "Orthogonal Defect Classification - A Concept for In-
Isabel Lopes Margarido, 6th of June 2012, Coimbra
Process Measurements," IEEE Transactions on Software Engineering,
vol. 18, pp. 943-956, November 1992.
 R. B. Grady, Practical software metrics for project management and
process improvement: Prentice-Hall, Inc., 1992.
 T. E. Bell and T. A. Thayer, "Software requirements: Are they really a
problem?," presented at the Proceedings of the 2nd international
conference on Software engineering, San Francisco, California, United
States, 1976.
 V. R. Basili and D. M. Weiss, "Evaluation of a software requirements
document by analysis of change data," presented at the Proceedings of
the 5th international conference on Software engineering, San Diego,
California, United States, 1981.
88
88/27

89. references
 G. S. Walia and J. C. Carver, "Development of Requirement Error
Taxonomy as a Quality Improvement Approach: A Systematic Literature
Review," Department of Computer Science and Engineering, 2007.
 A. F. Ackerman, et al., "Software Inspections: An Effective Verification
Process," IEEE Software, vol. 6, pp. 31-36, May 1989.
 A. A. Porter, et al., "Comparing Detection Methods for Software
Isabel Lopes Margarido, 6th of June 2012, Coimbra
Requirements Inspections: A Replicated Experiment," IEEE Transactions
on Software Engineering, vol. 21, pp. 563-575, June 1995.
 J. H. Hayes, et al., "Case History of International Space Station
Requirement Faults," presented at the Proceedings of the 11th IEEE
International Conference on Engineering of Complex Computer Systems,
Standford, California, 2006.
 M. Kalinowski, et al., "Applying DPPI: A Defect Causal Analysis Approach
Using Bayesian Networks," in Product-Focused Software Process
Improvement. vol. 6156, M. Ali Babar, et al., Eds., ed: Springer Berlin /
Heidelberg, 2010, pp. 92-106.
 M. Kalinowski, et al., "Guidance for Efficiently Implementing Defect Causal
Analysis," presented at the Brazilian Software Quality Symposium,VII
SBSQ Florianópolis, Brazil, 2008.
89
89/27

90. interested in our research?
published on IEEE:
 http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=597423
7&contentType=Conference+Publications&queryText%3DLopes+Marg
arido
Isabel Lopes Margarido, 6th of June 2012, Coimbra
author for correspondence:
 Isabel Lopes Margarido, isabel.margarido@gmail.com
 http://paginas.fe.up.pt/~pro09003/
affiliation
mui nobre
partially sponsored by:
90
90/27