This document summarizes a presentation on improving requirements quality. It discusses how poor quality requirements can impact projects and introduces requirements quality analysis. It then describes a proof of concept project conducted with a company to test a requirements quality suite tool. The project involved benchmarking tools, defining quality metrics, analyzing a specification, modifying requirements, and assessing the results. The proof of concept demonstrated the tool's ability to improve requirements quality and established next steps to further enhance requirements processes.

1. Jornada sobre
Innovación y Tendencias
en la Gestión de Requisitos
9 de mayo – Madrid
Sede Madrid Network
organizan:
#gestionrequisitos2016
Mejora en la calidad de requisitos:
Un caso de éxito
José M. Fuentes
1

2. Presenter profile
• José M. Fuentes
• jose.fuentes@reusecompany.com
• Co-founder of The REUSE Company
• Current Chief Commercial Officer of The REUSE Company
José M. Fuentes
CCO
• Member of AEIS Board (Spanish Chapter of
INCOSE)
• Member of INCOSE Requirements Engineering WG
• Contributor to INCOSE Guide for Writing
Requirements
• PM for TRC in EU Research Projects:
– AUTOSoft project
– CRYSTAL Project
– AMASS Project
– REVaMP2
2

3. The REUSE Company is a tool
manufacturer providing Knowledge-Centric
solutions to critical systems engineering
such as requirements quality management
or systems knowledge reuse.
The Requirements Quality Suite (RQS), a
tool to manage the requirements V&V
activity, is the most well-known product.
TRC is a SME based in Madrid (Spain)
Parque Tecnológico Legatec.
The REUSE Company profile
3

4. Contents
• Introduction
– The impact of poor quality in our projects
– Requirements Quality Analysis
• Practical case study
– Goals, inputs and expected outputs
– Tools benchmark
– The PoC Process
– PoC results
4

5. The impact of poor quality

6. The impact of poor quality projects
6

7. Why Requirements Quality Analysis?
7

8. Why Requirements Quality Analysis?
Doing the right thing right (verification)
http://www.theguardian.com/world/2014/may/21/french-railway-operator-sncf-orders-trains-
too-big
http://elpais.com/elpais/2015/02/04/inenglish/1423052376_326956.html
8

9. The FOCUS:
Requirements Quality Analysis

10. Why Requirements Quality Analysis?
Source : INCOSE SE Handbook V3.2
95%
85%
70%
Time
Cumulativepercentage
LifecylceCost
Operations
through
Disposal
100%
Production
and test
50%
8%
Design
15% 20%
Concept
Commited Costs
3-6x
500-1000x
20-100x
Development
10

11. Systems and Requirements
Engineering life-cycles
CONOPS
Stakeholders
Requirements
System
Requirements
System
Design
Equipment
Requirements
Equipment
Design Equipment
Verification
System
Equipment
SystemVerification
Product
Product Verification
11

12. Systems and Requirements
Engineering life-cycles
Elicitation Analysis Specification Validation
close gapsclarify
rewrite
re-evaluate
confirm and correct
Source: Karl Wiegers
12

13. Systems and Requirements
Engineering life-cycles
CONOPS
Stakeholders
Requirements
System
Requirements
System
Design
Equipment
Requirements
Equipment
Design Equipment
Verification
System
Equipment
SystemVerification
Product
Product Verification
Requirements
Verification
Requirements
Verification
Requirements
Verification
Design
Validation
Design
Verification
Requirements
Validation
13

14. Practical Application at

15. • Objectives
– Perform correctness, completeness and consistency analyses of
requirements (individually and collectively) to improve the Quality
of requirements specifications
– Assess the computer-aided requirements authoring feature to
accelerate the learning curve of new practitioners (or improve
the capability of current practitioners) in requirements
development
• Goal
– Exonerate engineers from format concerns (structure) and allow
them to concentrate on content (essence of requirements):
technical data useful for design
Quick Proof of Concept on Requirements
Quality Improvement
15

16. • External audits results
– “… Requirements Characterization is not complete:
Derived/uncovered requirements justification, Contribution,
Categories (technical vs non-technical), V&V Methods…
– …V&V Plan is not complete: Verification activities, or agreed
alternate practices (waivers) and associated deliverables…”
• CMMI for Development
– Requirement Development process area – SG 3 Analyze and
Validate Requirements
• “… Analyze requirements to determine whether they satisfy the of
higher level requirements.
Analyze requirements to ensure that they are complete, feasible,
realizable, and verifiable…”
– Verification process area – SG 2 Perform Peer Reviews
• “… Establish and maintain checklists to ensure that the work products
are reviewed consistently...
Rules of construction , Completeness, Correctness…”
Also, a means to improve current practices
16

17. Previous results
Most common requirements defects. Source: Gauthier Fanmuy - the RAMP project: - AFIS
17

18. Requirements quality characteristics vs
quality metrics
• Well-known requirements quality characteristics
• IEEE Std. 830:
– Correct
– Unambiguous
– Complete
– Consistent
– Ranked
– Verifiable
– Modifiable
– Traceable
• ESA PSS-05,
ISO/IEC 29148, others:
– Pretty much the same characteristics
• SMART:
– Specific
– Measurable
– Achievable
– Relevant
– Traceable
"I believe that this nation should commit
itself to achieving the goal, before this
decade is out, of landing a man on the
Moon and returning him safely to Earth"
18

19. Quality characteristics to measure
• How to… Perform CCC
Define Metrics for the Rules proposed in the
INCOSE Guide + Others
INCOSE Requirements Working Group
Guide for Writing Requirements V4
Characteristic Cxx – Characteristic name
Rationale: xxxx
Strategy: xxxx
Rules that help establish this characteristic:
Rxx - /Section/Rule name
Avoid xxxx
Ryy - /Section/Rule name
Avoid yyy
19

20. • Quality of individual requirements
– Correctness
 Requirement statement is understandable by humans (TRC)
 Requirement statement’s structure is agreed with the SE dept. (TRC)
 Requirement Statement corresponds to user request (ISO TR 24766)
 The requirement contains all the information necessary for design and
implementation (Alstom)
• Quality of requirements sets
– Consistency
 Absence of conflict among a set of requirements (ISO TR 24766)
 A set of requirements is consistent if each necessity (requirement) is
expressed in one and only one requirement (TRC)
– Completeness
 The set of requirements represents a complete definition of the product
(ISO TR 24766)
 Completeness by comparing project content vs project content (TRC)
Managing Requirements Quality: CCC approach
20

21. Requirements Quality Analysis Benchmark
INITIAL SOURCE:
• AFIS « RAMP » project 2010-2012
• Evaluations by AIRBUS 2012
21

22. Requirements Quality Suite
The Requirements Quality Suite (RQS) intends to tackle requirements quality
management by offering a set of tools and processes
Automatic measurement of requirements quality metric
Support to Requirements Authoring
RQS models requirements quality metrics using the CCC approach (Correctness,
Consistency and Completeness)
Requirements Quality Analyzer (RQA):
to setup, check and manage the quality of a
requirements specification.
Requirement Authoring Tool (RAT):
to assist authors while they are creating or editing
requirements.
Knowledge Manager (KM):
to manage knowledge around a requirements
specification: the ontology it is based on, the
structure of the requirements to be used in the
project, the communication between authors and
domain architects.22

23. • IBM DOORS ©
• PTC Integrity ©
• CATIA Reqtify ©
• OSLC
• VISURE Requirements ©
• Microsoft Excel ©
• XML file
Near future:
• Microsoft Word ©
• Siemens Teamcenter ©
RQS – Requirements Quality Suite: connectors
23

24. RQS – Requirements Quality Suite: languages
• RQS is highly dependent of the language of the
requirements
• Languages supported so far:
24

25. • Correctness metrics are quantitative
• Correctness metric values are calculated counting items
– Example: In the Metric Text length in words the metric counts the
number of words; then the QF transforms it into a quantitative value
• The process is simplified by using interval (step, discrete) quality functions
• Metrics use one of the following quality functions:
Managing the metrics Quality Functions (QF)
25
textLength()
Q
High
Med
Low
1 5 10 30 50

26. Quality Management Definition: Maturity
lifecycle
WHITE
Belt
Metrics
YELLOW
Belt
Metrics
ORANGE
Belt
Metrics
BLUE
Belt
Metrics
BROWN
Belt
Metrics
BLACK
Belt
Metrics
GREEN
Belt
Metrics
DEMANDING
LEVEL
textLength()
Q
High
Med
Low
1 5 10 30 50
textLength()
Q
High
Med
Low
1 4 6 2526

27. Specification quality assessment w.r.t. the
Maturity lifecycle
DEMANDING LEVEL
WHITE Belt
Metrics
YELLOW Belt
Metrics
ORANGE Belt
Metrics
GREEN Belt
Metrics
BROWN Belt
Metrics
BLACK Belt
Metrics
BLUE Belt
Metrics
CORRELATED LEVELS
OF RESULTS
27

28. Specification quality assessment: The Goal
WHITE Belt
Metrics
YELLOW Belt
Metrics
ORANGE Belt
Metrics
GREEN Belt
Metrics
BROWN Belt
Metrics
BLACK Belt
Metrics
BLUE Belt
Metrics
DEMANDING LEVEL
CORRELATED LEVELS
OF RESULTS
28

29. Specification quality assessment: The Path
TIME
WHITE Belt
Metrics
YELLOW Belt
Metrics
ORANGE Belt
Metrics
GREEN Belt
Metrics
BROWN Belt
Metrics
BLACK Belt
Metrics
BLUE Belt
Metrics
29

30. Specification quality assessment: Maturity level by depts.
or Teams
WHITE Belt
Metrics
YELLOW Belt
Metrics
ORANGE Belt
Metrics
GREEN Belt
Metrics
BROWN Belt
Metrics
BLACK Belt
Metrics
BLUE Belt
Metrics
TIME
30

31. The Knowledge Base
31
Terminology layer
Valid terms, forbidden terms, other NL
terms, Syntactic clustering types,
everything as concepts
Thesaurus layer
Relationships among concepts
(hierarchies, associations, synonyms…) as
well as semantic clusters and relationship
types
Patterns layer
Matching Patterns
Formalization layer
Semantic formalization
Inference layer
For decision making (e.g.
consistency, completeness)

32. • Patterns:
– Represents the structures every correct requirement should
meet
– Different types of requirements  different patterns (templates)
– Customizable for every domain, customer and content of each
requirements document
– Libraries with sets of patterns
– Represented as a sequential set of restrictions: placeholders
Examples of requirements metrics: patterns
32
When <Event> <Component> Shall <Action> <Object> Time_constraint

33. Knowledge Base: Example
33
A380 A350 System Operate Temperature Environment Pressure
Controlled
Vocabulary
A380 A350
<<Aircraft>> “ Greater than (>) “
Operate Work
<<Operation>>
<<Aircraft>> Shall <Operation> <<Minimum>>At Environment Of
[MEASUREMENT
UNIT]
NUMBER
temperature
“ Greater than (>) “
ºC-70
Patterns
Temperature Pressure
Environment
Temperature [-60ºC , +60ºC]
“ Operation Range “
Inference
Rules NUMBER “ Lower than (<) “ -60º NUMBER “ Greater than (>) “ +60º||
Thesaurus
Formalizations The aircraft shall be able to operate
at a minimum temperature of -70º C
If ºC ºC
“ Lower than (<) “
Shall
At a minimum
<<Minimum>>
At a minimum Of

34. • The REUSE Company has developed IT solutions that attempt to
understand, formalize, represent, reason-about and search-for all
kinds of knowledge assets
• Using: Terminology Control, Patterns, Graphs and Natural language
Processing
Knowledge Base: Requirements Patterns
matching
34
UR044 :The Radar shall identify hits at a minimum rate of 10 units per second
The <DEVICE> Shall <ACTION> <ITEMS> <MINIMUM>At Rate of
<RATE
VALUE>
[NUMBER]
Radar Hits
<<Detect>>
10 Units per Second
<<Minimum Value>>
Hits

35. • To promote requirements reuse
• To detect duplicates
• To provide quick access to related requirements
Knowledge base: semantic search engine
35

36. The PoC Process

37. Process: Work with one Requirements
Specification
37
Original
Reqs.
Specification
INITIAL QUALITY
ASSESSMENT
Original
Reqs.
Specification
QUALITY
METRICS
DEFINITION
Organization
Knowledge Base
V1
SPECIFICATION
UPDATE
Reqs.
Specification
Rev 1
Original
Reqs.
Specification
Organization
KB V1
Organization
KB V2
Quality
Results
Organization
KB V0
1st QUALITY
ASSESSMENT WITH
BELTS
Original
Reqs.
Specification
Organization
KB V1
2nd QUALITY
ASSESSMENT
WITH BELTS
Reqs.
Specification
Rev 1
Organization
KB V2

38. The PoC Results

39. Results: Initial Quality Assessment without belts
39
Original Specification
All out of the box metrics enabled 32 Metrics (INCOSE + TRC)
329 Requirements
% of Requirements

40. Effort centred in metrics for Correctness
Results: Developed Colored Belts
40
WHITE
Belt
YELLOW Belt ORANGE
Belt
20 Metrics correctness 31 Metrics correctness 51 Metrics correctness
+
2 Metrics completeness

41. Results: Developed Ontology
Terminology
Patterns
1 381 unclassified terms after first indexation
70 new organization specific patterns
41

42. Results: First Quality Assessment with belts
42
Original
Specification
329 Requirements
% of Requirements
WHITE Belt YELLOW Belt ORANGE Belt

43. Results: Specification modification (by experts)
43
WHITE
Belt
Metrics
329 Requirements 438 Requirements
66.26% Modified Reqs.
Final
Specification
% of Requirements % of Requirements
Original
Specification

44. Results: Comparison original vs modified
specification
44
Final Specification
Original Specification
White Belt Original vs. Final Spec.

45. • Achieved results
– A tool-supported requirements quality process
– A gradual set of quality metrics (belts)
– A Formal Reusable Knowledge Base
– ~70 new Patterns
– One improved Requirements Specification
– An Alstom “Guide for Requirements Authoring”
• Resources
– 2 participants from Alstom, 2 participants from TRC
– 1.5 effective months in calendar: 3.37 PM
 Learning and mastering of the tool suite
Definition of Metrics, Generation of Ontology
Modifications of requirements
Conclusion
45
0
50
100
150
200
250
White belt Yellow belt Orange belt
PoC Hours

46. REQUIREMENTS
QUALITY
ASSESSMENT
Specification
of Type 2
Customer
KB V2
Quality
Results
REQUIREMENTS
QUALITY
ASSESSMENT
Specification
of Type 3
Customer
KB V2
Quality
Results
• Decision to apply the white belt maturity to
analyze the quality of other specifications
– Fine-tune the metrics
• Decision to enhance the current Requirement
Engineering Process by including
requirements verification activities
• Decision to launch a real-scale pilot project:
real industrial project and staff, IT
infrastructure, measurement of performances
Conclusions of the PoC
46

47. Detalles de Contacto
José M. Fuentes
jose.fuentes@reusecompany.com
+34 912 17 25 96
@ReuseCompany
Mejora en la calidad de requisitos
47