Ambiguity Reviews: Building Quality Requirements


Published on

Are you frustrated by the false expectation that we can test quality into a product? By the time an application is delivered to testing, our ability to introduce quality principles is generally limited to defect detection. So how do you begin to shift your team’s perceptions into a true quality assurance organization? Susan Schanta shares her approach to Shift Quality Left by performing ambiguity reviews against requirements documents to reduce requirement defects at the beginning of the project. By helping the business analyst identify gaps in requirements, you can help build quality in and improve the team’s ability to write testable requirements. Learn how to review requirements to identify ambiguities and document the open questions that need to be addressed to make requirements clear, concise, and testable. Susan demonstrates her approach to ambiguity reviews and how she turned lessons learned into a Business Analyst Style Guide to drive quality into the requirements gathering process.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Ambiguity Reviews: Building Quality Requirements

  1. 1. T19 Requirements 5/8/2014 3:00:00 PM Ambiguity Reviews: Building Quality Requirements Presented by: Susan Schanta Cognizant Technology Solutions Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ ∙
  2. 2. Susan Schanta Cognizant Technology Solutions Susan Schanta has spent twenty years managing large-scale and quality assurance programs in both new ventures and global Fortune 500 companies in the financial, healthcare, and retail domains. Susan has led corporate initiatives in business optimization, onshore-to-offshore transitions, and QA enterprises including SDLC management, automation, and performance. Her expertise in industry-best practices and project execution has helped companies achieve their goals. Susan’s experience includes implementing lifecycle disciplines through change management, sharply reducing budget variances related to estimation techniques, improving performance of tiered applications, introducing new disciplines for UAT execution to achieve improved quality and business workflow processes, and devising calibration methods to measure these achievements.
  3. 3. 1 | © 2014, Cognizant0© 2014, Cognizant | All rights reserved. The information contained herein is subject to change without notice. Susan Schanta Process, Quality & Consulting April, 2014 Ambiguity Reviews Building Quality Requirements | © 2014, Cognizant1 Agenda The Cost of Quality 2 What is an Ambiguity Review? 3 Ambiguity Review Place in the SDLC 11 Ambiguity Review Classification 14 Ambiguity Review Types 20 Ambiguity Review Template 23 Ambiguity Review – Keyword Searches 27 Ambiguity Review – Content Review 30 Ambiguity Review Metrics 35 Ambiguity Review – Benefit to QA 40 Ambiguity Review – Be Nice About It! 42 Appendix 44
  4. 4. 2 | © 2014, Cognizant2 The Cost of Quality 2 • IRS Tax Systems modernization project spent $3.3b before canceling (Federal Computer Week March, 2002) • Time Warner Communications spent $1b on a failed information system to break into the Residential Telephone Business (Computerworld May, 1997) • Sainsbury, a British food retailer wrote off $526m invested in an Automated Supply-Chain Management System. • Ford Motor Company spent $400m on a Purchasing System before abandoning it in 2004 50% of defects are due to requirements problems (Schwaber, 2006) | © 2014, Cognizant3 What is an Ambiguity Review?
  5. 5. 3 | © 2014, Cognizant4 What is an Ambiguity? Ambiguity can be defined as 1. Use of words that allow alternative interpretations 2. An unclear, indefinite, or equivocal word, expression, meaning, etc. 3. The possibility of interpreting an expression in two or more distinct ways 4. Doubtfulness or uncertainty of meaning or intention: to speak with ambiguity; an ambiguity of manner. ( | © 2014, Cognizant5 What is an Ambiguity Review? An Ambiguity Review is a formal review process that focuses on the identification of ambiguities in the language, structure and logic of a requirement. − Provides a measure of whether requirements are quantitative, clear, correct and complete. − Eliminates requirements defects from being identified in later phases by building quality into the product. (Bender)
  6. 6. 4 | © 2014, Cognizant6 What Are the Benefits of Ambiguity Reviews? Ambiguity Benefits realized • Scope Creep is controlled • Reduced cost of maintenance • Reduced number of change requests • Increased traceability from requirements to test cases −Defects per requirement Tangible Results Healthcare Insurer • Reduced Defect Leakage from 35% to 8% • Introduced BA Style Guide to drive down ambiguities Fortune 50 Insurance Company #1 • Reduced ambiguities from an avg. 7 per Req. to 2 per Req. • Introduced BA Style Guide to drive down ambiguities Fortune 50 Insurance Company #2 • 63% of all defects were traced to Requirement Defects | © 2014, Cognizant7 Business Users Focus on Happy Path Impact to business users • Who focus on Happy Path Results identify • Open ended questions What are your pain points? • Requirements lack input/output, alternative flows, error conditions and constraints • Development makes assumptions to fills in gaps in the requirements • Potential Defect leakage to production Why do an Ambiguity Review?
  7. 7. 5 | © 2014, Cognizant8 Development Drives Requirements Impact to Development • Who drive application behavior based on assumptions where requirement gaps exist Results identify • Higher maintenance is a higher percentage of IT budget − Limits dollars spent on revenue producing software − Increases redo work • Change requests trend higher • Quality Assurance is expected to test quality in Why do an Ambiguity Review? | © 2014, Cognizant9 QA Builds In Quality, Right? Impact to QA • Who are expected to test quality into the product Quality cannot be tested into the product Results identify • Test Cases may not address.. − Gaps in user scenarios − Gaps in alternative flows (error conditions) − Gaps in business rules • Confusion and delays with requirements often results in a compressed schedule for testing − Increases redo work • Defect leakage into production Why do an Ambiguity Review?
  8. 8. 6 | © 2014, Cognizant10 Ambiguity Review Benefits • Limit Scope Creep • Increase Requirements Traceability to Test Cases • Reduce Defect Leakage to Production • Improves Estimation Accuracy • Reduce Cost of Maintenance − How much is your organization spending to maintain systems today? • Reduces redo work • Increases velocity of Test Case creation • Increase BA productivity and work product − Elicitation Checklists − BA Style Guide How do I measure success? | © 2014, Cognizant11 Ambiguity Review Place in the SDLC
  9. 9. 7 | © 2014, Cognizant12 Ambiguity Reviews in Requirements Phase Requirements Phase • Elicitation Preparation • Elicitation Session • Document Elicitation Results • Confirm Elicitation Results − BA Peer Review − Ambiguity Review − Final Review & Signoff by Key Stakeholders Note: Ambiguity Reviews are SDLC agnostic. The process can be applied to any lifecycle and any document format. Building Quality In | © 2014, Cognizant13 Ambiguity Review in the Requirements Phase Where do Ambiguity Reviews fit into the Lifecycle?
  10. 10. 8 | © 2014, Cognizant14 Ambiguity Review Classification | © 2014, Cognizant15 Ambiguity of Reference & Ambiguous Statements A condition when a requirement uses words such as pronouns, adjectives, adverbs and verbs that can be interpreted differently based on the reader’s view. • The report shall run frequently. Ambiguity: What is the name of the report? Ambiguity: What are the data elements in the report? Ambiguity: Frequently is not measurable Ambiguity: What is the report generation schedule? Ambiguity: What happens if there is no data for the report? Ambiguity: What happens if the report fails? (Wiegers, Bender)
  11. 11. 9 | © 2014, Cognizant16 Boundary Ambiguity A condition when the author uses terms - among or up to. The scope of the requirement is ambiguous because the stated requirement can be interpreted in multiple ways. Example: 1. If an employee makes less than $20,000 per year, the employer pays 100% of the healthcare premium. 2. If an employee makes more than $20,000 per year, the employer pays 50% of the healthcare premium for the employee. Ambiguity: What if an employee makes exactly $20,000? (Wiegers, Bender) | © 2014, Cognizant17 Built-In Assumptions A condition when the author assumes that all consumers of the document will have the same level of domain knowledge • Industry domain knowledge • Subject domain knowledge • Functional knowledge • Environmental knowledge Example: The system must apply the same limitations to searches for existing groups as currently exists in Google Search. Ambiguity: The requirement assumes the reader knows how the functionality exists today. (Wiegers, Bender)
  12. 12. 10 | © 2014, Cognizant18 Dangling Else A condition when a requirement states expected results (what normally happens) but does not state exceptions and error conditions. Dangling Else Can Shall Could Should Is one of Will Must Would Example: The employee address type shall be either house, apartment or condominium. Ambiguity: The requirement does not consider exception conditions such as PO Box. (Wiegers, Bender) | © 2014, Cognizant19 Etc. Etcetera is not a quantifiable measurement that can be confirmed so it is considered totally ambiguous. (Phrases or sentences ending with etc ) Example: Subscribers shall identify themselves with unique information (policy number, social security, etc.) when they call Customer Care for information about their policy. (Wiegers, Bender)
  13. 13. 11 | © 2014, Cognizant20 Ambiguity Review Types | © 2014, Cognizant21 Categorizing Ambiguities to Support Metrics Ambiguity Categorizations will help with • Tracking patterns of ambiguities − For a Business Analyst − For the Program • Building Elicitation Checklists − Lessons learned turned into questionnaires − Mentoring sessions for the BA Team • Team Performance Measurements − Scope Management − Defect Leakage to Production − Requirement Defects Purpose of Ambiguity Types
  14. 14. 12 | © 2014, Cognizant22 Ambiguity Type Description Ambiguous Term Terms (Phrase or Word) used in requirements which can be interpreted by the reader in multiple ways e.g. frequently, occasionally, efficiently Conflicting Requirement Requirements which contradict each other – either in the same document or across multiple documents. e.g. The field name is Effective Date but the data type is defined as an integer Glossary Word or acronym used in requirements that is new or not commonly used but has not been defined in the Glossary/Definitions section. Grammar, Spelling & Wording Grammar, spelling corrections and proposed wording improvements to increase clarity of the requirement Incomplete Requirement Incomplete requirement or statement describing conditions when information is not fully detailed preventing design or test validation e.g. The system shall handle 15-25% increase in the second year Missing Requirement Missing requirements that were not documented or may not have been elicited from the business user. e.g. Missing requirements – alternative flows, business rules, exceptions and error conditions (Questions – What, When, Where) Unclear Requirement Requirements or statements requiring further clarification to allow the reader user to fully understand the requirement (Questions – How, Why, What do you mean by ) (Wiegers, Bender) | © 2014, Cognizant23 Ambiguity Review Template
  15. 15. 13 | © 2014, Cognizant24 Ambiguity Review Template • Transfer requirements document to Excel template − Revision History − Introductory information (free text) − Business Requirements − Data presented in tables such as Glossary, Field Elements and Financial Information segregated to its own worksheet • Ambiguity Columns to the Right − Ambiguity Type − Ambiguity Description Transform the Requirements Document to Excel | © 2014, Cognizant25 Requirements Worksheet Example Not all columns need to be transferred to the Ambiguity Template
  16. 16. 14 | © 2014, Cognizant26 Glossary Worksheet Example Text transferred to Excel template | © 2014, Cognizant27 Ambiguity Review – Keyword Searches
  17. 17. 15 | © 2014, Cognizant28 Ambiguity Review Keywords An initial review for keywords helps to identify Incomplete Requirements All files are transmitted daily. All 820 Payments Files shall be transmitted to the Federal and Maryland Jurisdiction after the nightly batch jobs complete • The Federal 820 Payment File shall be sent daily during the transmission window between 11 PM and 4 AM EST • The Maryland 820 Payment File shall be sent daily during the transmission window between Midnight and 4 AM EST • If no payments are made for a jurisdiction, an empty file shall be sent to the jurisdiction Clues to Finding Ambiguities | © 2014, Cognizant29 Ambiguity of Reference Ambiguous Adjectives Ambiguous Adverbs Ambiguous Variables Ambiguous Verbs above ordinary infrequently the database derive below rare intuitively the field determine it same just about the file edit such seamless more often than not the frame enable the previous several more or less the information improve them similar mostly the message indicate these some nearly the module manipulate they standard normally the page match this the complete not quite the rule maximize those the entire often the screen may Ambiguous Adjectives transparent on the odd occasion the status might all typical ordinarily the system minimize any usual rarely the table modify appropriate valid roughly the value optimize custom Ambiguous Adverbs seamlessly the window perform efficient accordingly seldom Ambiguous Verbs process every almost similarly adjust produce few approximately sometime alter provide frequent by and large somewhat amend support improved commonly transparently calculate update infrequent customarily typically change validate intuitive efficiently usually compare verify invalid frequently Ambiguous Variables compute many generally the application convert most hardly ever the component create normal in general the data customize (Bender/Wiegers)
  18. 18. 16 | © 2014, Cognizant30 Ambiguity Review – Content Review | © 2014, Cognizant31 Ambiguity Review for Content • Ambiguities in content focus on the following − Conflicting Requirements − Incomplete Requirements − Missing Requirements − Unclear Requirements • Questions of Who, What, When, Where & Why • Questions of How tend toward design questions − Only use to clarify expected behavior or outcome − Do not use to ask how the system will process the function behind the scenes Clues to Finding Ambiguities
  19. 19. 17 | © 2014, Cognizant32 Sample Ambiguity Questions • What is the name of the file? • Who is the user? • What permissions does the user need to review the file? • When is the file sent? • What are the contents of the file? • Where is the file sent? • What are the business rules to validate the file? • What are the error messages displayed when a business rule validation fails? • What happens if the file is corrupted and can’t be read? • What happens if the file fails to be generated? • Is an alert sent if a file is late, corrupt or fails? • Who is the alert sent to? Who, What, When, Where, Why | © 2014, Cognizant33 Ambiguity Review Example If the policy is a subscriber + spouse/domestic partner + dependents policy, subscriber wants to cancel from the policy, Exchange will send us 834 file to enroll the spouse/ domestic partner as the subscriber under a new SID and the dependents will remain as dependents under the new subscriber. Note: Subscriber + Family Content Questions Raised for this Requirement
  20. 20. 18 | © 2014, Cognizant34 Ambiguity Questions Posed | © 2014, Cognizant35 Ambiguity Review Metrics
  21. 21. 19 | © 2014, Cognizant36 Ambiguity Review Metrics Patterns within the Requirements Document 3 6 Top 10 Ambiguities by Type Total Ambiguous Term 55 Grammar Spelling & Wording 79 Incomplete Requirements 59 Unclear Requirements 57 Total Ambiguities 250 | © 2014, Cognizant37 Program Level Ambiguity Review Metrics Patterns across the Requirements Document
  22. 22. 20 | © 2014, Cognizant38 Program Level Ambiguity Review Metrics Patterns across the Requirements Document | © 2014, Cognizant39 Program Level Ambiguity Review Metrics Patterns across the Requirements Document Ambiguity Review Summary Totals Total Requirement Considered* 3,737 Total Ambiguous Observations 26,478 Top 10 Ambiguities 7,463 Percent Ambiguities in Top Ten 28% Average Ambiguities Per Requirement 7 *Requirements include Non-Requirement Sections such as Introduction, Purpose, etc.
  23. 23. 21 | © 2014, Cognizant40 Ambiguity Review – Benefit to QA | © 2014, Cognizant41 QA Benefits from Ambiguity Review • Geographically dispersed teams benefit from the additional level of detail documented in the requirements • Test Case Creation is simplified by clear requirements − Normal Flow & Alternative Flows detailed in Use Cases − Error conditions with error messages − Business rules and constraints − Data requirements − Expected Outcome • Traceability to requirements provides QA Engineer with path to use cases and functional/non functional requirements − Diagnosis of defect types clear • Requirements vs. Code vs. Test Case Quantifiable Impact
  24. 24. 22 | © 2014, Cognizant42 Ambiguity Review – Be Nice About It! | © 2014, Cognizant43 Business Analysts Have Feelings Ambiguity Reviews are meant to be objective and focus on the requirements content. However You are critiquing a Business Analyst’s work. • Recognize the Business Analyst’s work effort • Recognize the Business Analyst’s challenges with eliciting requirements from business users. • Be courteous when providing feedback. Write Ambiguity Questions that are quantitative, clear, correct and complete! Be respectful of your feedback
  25. 25. 23 | © 2014, Cognizant44 Appendix | © 2014, Cognizant45 Requirements Review Phase
  26. 26. 24 | © 2014, Cognizant46 References Writing High Quality Requirements By Karl E. Wiegers, 2006 The Ambiguity Review Process By Richard Bender Assessing the Impact of Poor Requirements on Companies By Keith Ellis Executive Guide to Evaluating Business Requirements Quality By Keith Ellis Getting Consensus on Business Requirements – Tips and Traps By Keith Ellis The Quest for Good Requirements By Dr. Martin Schedlbauer, 2011 How to Prevent the Negative Impacts of Poor Requirements By Sergey Korban, April 30, 2013 The Business Value of Better Requirements By Karl E. Wiegers, 2006