Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Requirements' Quality Improvement: A Successful Case Study


Published on

Low quality in our requirements specifications impact, significantly, in the overall quality of the projects, causing important losses as delays, loss of contracts, and in some rare cases, losses related to the environment or even human lives. A successful case study focused on improving processes and techniques will be presented as well as tools related to authoring, verification and validation of requirements.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Requirements' Quality Improvement: A Successful Case Study

  1. 1. Jornada sobre Innovación y Tendencias en la Gestión de Requisitos 9 de mayo – Madrid Sede Madrid Network organizan: #gestionrequisitos2016 Mejora en la calidad de requisitos: Un caso de éxito José M. Fuentes 1
  2. 2. Presenter profile • José M. Fuentes • • Co-founder of The REUSE Company • Current Chief Commercial Officer of The REUSE Company José M. Fuentes CCO • Member of AEIS Board (Spanish Chapter of INCOSE) • Member of INCOSE Requirements Engineering WG • Contributor to INCOSE Guide for Writing Requirements • PM for TRC in EU Research Projects: – AUTOSoft project – CRYSTAL Project – AMASS Project – REVaMP2 2
  3. 3. The REUSE Company is a tool manufacturer providing Knowledge-Centric solutions to critical systems engineering such as requirements quality management or systems knowledge reuse. The Requirements Quality Suite (RQS), a tool to manage the requirements V&V activity, is the most well-known product. TRC is a SME based in Madrid (Spain) Parque Tecnológico Legatec. The REUSE Company profile 3
  4. 4. Contents • Introduction – The impact of poor quality in our projects – Requirements Quality Analysis • Practical case study – Goals, inputs and expected outputs – Tools benchmark – The PoC Process – PoC results 4
  5. 5. The impact of poor quality
  6. 6. The impact of poor quality projects 6
  7. 7. Why Requirements Quality Analysis? 7
  8. 8. Why Requirements Quality Analysis? Doing the right thing right (verification) too-big 8
  9. 9. The FOCUS: Requirements Quality Analysis
  10. 10. Why Requirements Quality Analysis? Source : INCOSE SE Handbook V3.2 95% 85% 70% Time Cumulativepercentage LifecylceCost Operations through Disposal 100% Production and test 50% 8% Design 15% 20% Concept Commited Costs 3-6x 500-1000x 20-100x Development 10
  11. 11. Systems and Requirements Engineering life-cycles CONOPS Stakeholders Requirements System Requirements System Design Equipment Requirements Equipment Design Equipment Verification System Equipment SystemVerification Product Product Verification 11
  12. 12. Systems and Requirements Engineering life-cycles Elicitation Analysis Specification Validation close gapsclarify rewrite re-evaluate confirm and correct Source: Karl Wiegers 12
  13. 13. Systems and Requirements Engineering life-cycles CONOPS Stakeholders Requirements System Requirements System Design Equipment Requirements Equipment Design Equipment Verification System Equipment SystemVerification Product Product Verification Requirements Verification Requirements Verification Requirements Verification Design Validation Design Verification Requirements Validation 13
  14. 14. Practical Application at
  15. 15. • Objectives – Perform correctness, completeness and consistency analyses of requirements (individually and collectively) to improve the Quality of requirements specifications – Assess the computer-aided requirements authoring feature to accelerate the learning curve of new practitioners (or improve the capability of current practitioners) in requirements development • Goal – Exonerate engineers from format concerns (structure) and allow them to concentrate on content (essence of requirements): technical data useful for design Quick Proof of Concept on Requirements Quality Improvement 15
  16. 16. • External audits results – “… Requirements Characterization is not complete: Derived/uncovered requirements justification, Contribution, Categories (technical vs non-technical), V&V Methods… – …V&V Plan is not complete: Verification activities, or agreed alternate practices (waivers) and associated deliverables…” • CMMI for Development – Requirement Development process area – SG 3 Analyze and Validate Requirements • “… Analyze requirements to determine whether they satisfy the of higher level requirements. Analyze requirements to ensure that they are complete, feasible, realizable, and verifiable…” – Verification process area – SG 2 Perform Peer Reviews • “… Establish and maintain checklists to ensure that the work products are reviewed consistently... Rules of construction , Completeness, Correctness…” Also, a means to improve current practices 16
  17. 17. Previous results Most common requirements defects. Source: Gauthier Fanmuy - the RAMP project: - AFIS 17
  18. 18. Requirements quality characteristics vs quality metrics • Well-known requirements quality characteristics • IEEE Std. 830: – Correct – Unambiguous – Complete – Consistent – Ranked – Verifiable – Modifiable – Traceable • ESA PSS-05, ISO/IEC 29148, others: – Pretty much the same characteristics • SMART: – Specific – Measurable – Achievable – Relevant – Traceable "I believe that this nation should commit itself to achieving the goal, before this decade is out, of landing a man on the Moon and returning him safely to Earth" 18
  19. 19. Quality characteristics to measure • How to… Perform CCC Define Metrics for the Rules proposed in the INCOSE Guide + Others INCOSE Requirements Working Group Guide for Writing Requirements V4 Characteristic Cxx – Characteristic name Rationale: xxxx Strategy: xxxx Rules that help establish this characteristic: Rxx - /Section/Rule name Avoid xxxx Ryy - /Section/Rule name Avoid yyy 19
  20. 20. • Quality of individual requirements – Correctness  Requirement statement is understandable by humans (TRC)  Requirement statement’s structure is agreed with the SE dept. (TRC)  Requirement Statement corresponds to user request (ISO TR 24766)  The requirement contains all the information necessary for design and implementation (Alstom) • Quality of requirements sets – Consistency  Absence of conflict among a set of requirements (ISO TR 24766)  A set of requirements is consistent if each necessity (requirement) is expressed in one and only one requirement (TRC) – Completeness  The set of requirements represents a complete definition of the product (ISO TR 24766)  Completeness by comparing project content vs project content (TRC) Managing Requirements Quality: CCC approach 20
  21. 21. Requirements Quality Analysis Benchmark INITIAL SOURCE: • AFIS « RAMP » project 2010-2012 • Evaluations by AIRBUS 2012 21
  22. 22. Requirements Quality Suite The Requirements Quality Suite (RQS) intends to tackle requirements quality management by offering a set of tools and processes Automatic measurement of requirements quality metric Support to Requirements Authoring RQS models requirements quality metrics using the CCC approach (Correctness, Consistency and Completeness) Requirements Quality Analyzer (RQA): to setup, check and manage the quality of a requirements specification. Requirement Authoring Tool (RAT): to assist authors while they are creating or editing requirements. Knowledge Manager (KM): to manage knowledge around a requirements specification: the ontology it is based on, the structure of the requirements to be used in the project, the communication between authors and domain architects.22
  23. 23. • IBM DOORS © • PTC Integrity © • CATIA Reqtify © • OSLC • VISURE Requirements © • Microsoft Excel © • XML file Near future: • Microsoft Word © • Siemens Teamcenter © RQS – Requirements Quality Suite: connectors 23
  24. 24. RQS – Requirements Quality Suite: languages • RQS is highly dependent of the language of the requirements • Languages supported so far: 24
  25. 25. • Correctness metrics are quantitative • Correctness metric values are calculated counting items – Example: In the Metric Text length in words the metric counts the number of words; then the QF transforms it into a quantitative value • The process is simplified by using interval (step, discrete) quality functions • Metrics use one of the following quality functions: Managing the metrics Quality Functions (QF) 25 textLength() Q High Med Low 1 5 10 30 50
  26. 26. Quality Management Definition: Maturity lifecycle WHITE Belt Metrics YELLOW Belt Metrics ORANGE Belt Metrics BLUE Belt Metrics BROWN Belt Metrics BLACK Belt Metrics GREEN Belt Metrics DEMANDING LEVEL textLength() Q High Med Low 1 5 10 30 50 textLength() Q High Med Low 1 4 6 2526
  27. 27. Specification quality assessment w.r.t. the Maturity lifecycle DEMANDING LEVEL WHITE Belt Metrics YELLOW Belt Metrics ORANGE Belt Metrics GREEN Belt Metrics BROWN Belt Metrics BLACK Belt Metrics BLUE Belt Metrics CORRELATED LEVELS OF RESULTS 27
  28. 28. Specification quality assessment: The Goal WHITE Belt Metrics YELLOW Belt Metrics ORANGE Belt Metrics GREEN Belt Metrics BROWN Belt Metrics BLACK Belt Metrics BLUE Belt Metrics DEMANDING LEVEL CORRELATED LEVELS OF RESULTS 28
  29. 29. Specification quality assessment: The Path TIME WHITE Belt Metrics YELLOW Belt Metrics ORANGE Belt Metrics GREEN Belt Metrics BROWN Belt Metrics BLACK Belt Metrics BLUE Belt Metrics 29
  30. 30. Specification quality assessment: Maturity level by depts. or Teams WHITE Belt Metrics YELLOW Belt Metrics ORANGE Belt Metrics GREEN Belt Metrics BROWN Belt Metrics BLACK Belt Metrics BLUE Belt Metrics TIME 30
  31. 31. The Knowledge Base 31 Terminology layer Valid terms, forbidden terms, other NL terms, Syntactic clustering types, everything as concepts Thesaurus layer Relationships among concepts (hierarchies, associations, synonyms…) as well as semantic clusters and relationship types Patterns layer Matching Patterns Formalization layer Semantic formalization Inference layer For decision making (e.g. consistency, completeness)
  32. 32. • Patterns: – Represents the structures every correct requirement should meet – Different types of requirements  different patterns (templates) – Customizable for every domain, customer and content of each requirements document – Libraries with sets of patterns – Represented as a sequential set of restrictions: placeholders Examples of requirements metrics: patterns 32 When <Event> <Component> Shall <Action> <Object> Time_constraint
  33. 33. Knowledge Base: Example 33 A380 A350 System Operate Temperature Environment Pressure Controlled Vocabulary A380 A350 <<Aircraft>> “ Greater than (>) “ Operate Work <<Operation>> <<Aircraft>> Shall <Operation> <<Minimum>>At Environment Of [MEASUREMENT UNIT] NUMBER temperature “ Greater than (>) “ ºC-70 Patterns Temperature Pressure Environment Temperature [-60ºC , +60ºC] “ Operation Range “ Inference Rules NUMBER “ Lower than (<) “ -60º NUMBER “ Greater than (>) “ +60º|| Thesaurus Formalizations The aircraft shall be able to operate at a minimum temperature of -70º C If ºC ºC “ Lower than (<) “ Shall At a minimum <<Minimum>> At a minimum Of
  34. 34. • The REUSE Company has developed IT solutions that attempt to understand, formalize, represent, reason-about and search-for all kinds of knowledge assets • Using: Terminology Control, Patterns, Graphs and Natural language Processing Knowledge Base: Requirements Patterns matching 34 UR044 :The Radar shall identify hits at a minimum rate of 10 units per second The <DEVICE> Shall <ACTION> <ITEMS> <MINIMUM>At Rate of <RATE VALUE> [NUMBER] Radar Hits <<Detect>> 10 Units per Second <<Minimum Value>> Hits
  35. 35. • To promote requirements reuse • To detect duplicates • To provide quick access to related requirements Knowledge base: semantic search engine 35
  36. 36. The PoC Process
  37. 37. Process: Work with one Requirements Specification 37 Original Reqs. Specification INITIAL QUALITY ASSESSMENT Original Reqs. Specification QUALITY METRICS DEFINITION Organization Knowledge Base V1 SPECIFICATION UPDATE Reqs. Specification Rev 1 Original Reqs. Specification Organization KB V1 Organization KB V2 Quality Results Organization KB V0 1st QUALITY ASSESSMENT WITH BELTS Original Reqs. Specification Organization KB V1 2nd QUALITY ASSESSMENT WITH BELTS Reqs. Specification Rev 1 Organization KB V2
  38. 38. The PoC Results
  39. 39. Results: Initial Quality Assessment without belts 39 Original Specification All out of the box metrics enabled 32 Metrics (INCOSE + TRC) 329 Requirements % of Requirements
  40. 40. Effort centred in metrics for Correctness Results: Developed Colored Belts 40 WHITE Belt YELLOW Belt ORANGE Belt 20 Metrics correctness 31 Metrics correctness 51 Metrics correctness + 2 Metrics completeness
  41. 41. Results: Developed Ontology Terminology Patterns 1 381 unclassified terms after first indexation 70 new organization specific patterns 41
  42. 42. Results: First Quality Assessment with belts 42 Original Specification 329 Requirements % of Requirements WHITE Belt YELLOW Belt ORANGE Belt
  43. 43. Results: Specification modification (by experts) 43 WHITE Belt Metrics 329 Requirements 438 Requirements 66.26% Modified Reqs. Final Specification % of Requirements % of Requirements Original Specification
  44. 44. Results: Comparison original vs modified specification 44 Final Specification Original Specification White Belt Original vs. Final Spec.
  45. 45. • Achieved results – A tool-supported requirements quality process – A gradual set of quality metrics (belts) – A Formal Reusable Knowledge Base – ~70 new Patterns – One improved Requirements Specification – An Alstom “Guide for Requirements Authoring” • Resources – 2 participants from Alstom, 2 participants from TRC – 1.5 effective months in calendar: 3.37 PM  Learning and mastering of the tool suite Definition of Metrics, Generation of Ontology Modifications of requirements Conclusion 45 0 50 100 150 200 250 White belt Yellow belt Orange belt PoC Hours
  46. 46. REQUIREMENTS QUALITY ASSESSMENT Specification of Type 2 Customer KB V2 Quality Results REQUIREMENTS QUALITY ASSESSMENT Specification of Type 3 Customer KB V2 Quality Results • Decision to apply the white belt maturity to analyze the quality of other specifications – Fine-tune the metrics • Decision to enhance the current Requirement Engineering Process by including requirements verification activities • Decision to launch a real-scale pilot project: real industrial project and staff, IT infrastructure, measurement of performances Conclusions of the PoC 46
  47. 47. Detalles de Contacto José M. Fuentes +34 912 17 25 96 @ReuseCompany Mejora en la calidad de requisitos 47