Are you frustrated by the false expectation that we can test quality into a product? By the time an application is delivered to testing, our ability to introduce quality principles is generally limited to defect detection. So how do you begin to shift your team’s perceptions into a true quality assurance organization? Susan Schanta shares her approach to Shift Quality Left by performing ambiguity reviews against requirements documents to reduce requirement defects at the beginning of the project. By helping the business analyst identify gaps in requirements, you can help build quality in and improve the team’s ability to write testable requirements. Learn how to review requirements to identify ambiguities and document the open questions that need to be addressed to make requirements clear, concise, and testable. Susan demonstrates her approach to ambiguity reviews and how she turned lessons learned into a Business Analyst Style Guide to drive quality into the requirements gathering process.

1. T19
Requirements
5/8/2014 3:00:00 PM
Ambiguity Reviews: Building
Quality Requirements
Presented by:
Susan Schanta
Cognizant Technology Solutions
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Susan Schanta
Cognizant Technology Solutions
Susan Schanta has spent twenty years managing large-scale and quality assurance programs
in both new ventures and global Fortune 500 companies in the financial, healthcare, and retail
domains. Susan has led corporate initiatives in business optimization, onshore-to-offshore
transitions, and QA enterprises including SDLC management, automation, and performance.
Her expertise in industry-best practices and project execution has helped companies achieve
their goals. Susan’s experience includes implementing lifecycle disciplines through change
management, sharply reducing budget variances related to estimation techniques, improving
performance of tiered applications, introducing new disciplines for UAT execution to achieve
improved quality and business workflow processes, and devising calibration methods to
measure these achievements.

3. 1
| © 2014, Cognizant0© 2014, Cognizant | All rights reserved. The information contained herein is subject to change without notice.
Susan Schanta
Process, Quality & Consulting
April, 2014
Ambiguity Reviews
Building Quality Requirements
| © 2014, Cognizant1
Agenda
The Cost of Quality 2
What is an Ambiguity Review? 3
Ambiguity Review Place in the SDLC 11
Ambiguity Review Classification 14
Ambiguity Review Types 20
Ambiguity Review Template 23
Ambiguity Review – Keyword Searches 27
Ambiguity Review – Content Review 30
Ambiguity Review Metrics 35
Ambiguity Review – Benefit to QA 40
Ambiguity Review – Be Nice About It! 42
Appendix 44

4. 2
| © 2014, Cognizant2
The Cost of Quality
2
• IRS Tax Systems modernization project spent $3.3b before
canceling (Federal Computer Week March, 2002)
• Time Warner Communications spent $1b on a failed
information system to break into the Residential Telephone
Business (Computerworld May, 1997)
• Sainsbury, a British food retailer wrote off $526m
invested in an Automated Supply-Chain Management
System.
• Ford Motor Company spent $400m on a Purchasing
System before abandoning it in 2004
50% of defects are due to requirements problems
(Schwaber, 2006)
| © 2014, Cognizant3
What is an Ambiguity Review?

5. 3
| © 2014, Cognizant4
What is an Ambiguity?
Ambiguity can be defined as
1. Use of words that allow alternative interpretations
2. An unclear, indefinite, or equivocal word, expression,
meaning, etc.
3. The possibility of interpreting an expression in two or more
distinct ways
4. Doubtfulness or uncertainty of meaning or intention: to speak
with ambiguity; an ambiguity of manner. (dictionary.com)
| © 2014, Cognizant5
What is an Ambiguity Review?
An Ambiguity Review is a formal review process that focuses
on the identification of ambiguities in the language, structure
and logic of a requirement.
− Provides a measure of whether requirements are
quantitative, clear, correct and complete.
− Eliminates requirements defects from being identified in
later phases by building quality into the product. (Bender)

6. 4
| © 2014, Cognizant6
What Are the Benefits of Ambiguity Reviews?
Ambiguity Benefits realized
• Scope Creep is controlled
• Reduced cost of maintenance
• Reduced number of change requests
• Increased traceability from requirements to test cases
−Defects per requirement
Tangible Results
Healthcare Insurer
• Reduced Defect Leakage from 35% to 8%
• Introduced BA Style Guide to drive down ambiguities
Fortune 50 Insurance Company #1
• Reduced ambiguities from an avg. 7 per Req. to 2 per Req.
• Introduced BA Style Guide to drive down ambiguities
Fortune 50 Insurance Company #2
• 63% of all defects were traced to Requirement Defects
| © 2014, Cognizant7
Business Users Focus on Happy Path
Impact to business users
• Who focus on Happy Path
Results identify
• Open ended questions
What are your pain points?
• Requirements lack input/output, alternative flows, error
conditions and constraints
• Development makes assumptions to fills in gaps in the
requirements
• Potential Defect leakage to production
Why do an Ambiguity Review?

7. 5
| © 2014, Cognizant8
Development Drives Requirements
Impact to Development
• Who drive application behavior based on assumptions
where requirement gaps exist
Results identify
• Higher maintenance is a higher percentage of IT budget
− Limits dollars spent on revenue producing software
− Increases redo work
• Change requests trend higher
• Quality Assurance is expected to test quality in
Why do an Ambiguity Review?
| © 2014, Cognizant9
QA Builds In Quality, Right?
Impact to QA
• Who are expected to test quality into the product
Quality cannot be tested into the product
Results identify
• Test Cases may not address..
− Gaps in user scenarios
− Gaps in alternative flows (error conditions)
− Gaps in business rules
• Confusion and delays with requirements often results in a
compressed schedule for testing
− Increases redo work
• Defect leakage into production
Why do an Ambiguity Review?

8. 6
| © 2014, Cognizant10
Ambiguity Review Benefits
• Limit Scope Creep
• Increase Requirements Traceability to Test Cases
• Reduce Defect Leakage to Production
• Improves Estimation Accuracy
• Reduce Cost of Maintenance
− How much is your organization spending to maintain systems today?
• Reduces redo work
• Increases velocity of Test Case creation
• Increase BA productivity and work product
− Elicitation Checklists
− BA Style Guide
How do I measure success?
| © 2014, Cognizant11
Ambiguity Review Place in the SDLC

9. 7
| © 2014, Cognizant12
Ambiguity Reviews in Requirements Phase
Requirements Phase
• Elicitation Preparation
• Elicitation Session
• Document Elicitation Results
• Confirm Elicitation Results
− BA Peer Review
− Ambiguity Review
− Final Review & Signoff by Key Stakeholders
Note: Ambiguity Reviews are SDLC agnostic. The process
can be applied to any lifecycle and any document format.
Building Quality In
| © 2014, Cognizant13
Ambiguity Review in the Requirements Phase
Where do Ambiguity Reviews fit into the Lifecycle?

10. 8
| © 2014, Cognizant14
Ambiguity Review Classification
| © 2014, Cognizant15
Ambiguity of Reference & Ambiguous Statements
A condition when a requirement uses words such as pronouns,
adjectives, adverbs and verbs that can be interpreted differently
based on the reader’s view.
• The report shall run frequently.
Ambiguity: What is the name of the report?
Ambiguity: What are the data elements in the report?
Ambiguity: Frequently is not measurable
Ambiguity: What is the report generation schedule?
Ambiguity: What happens if there is no data for the report?
Ambiguity: What happens if the report fails?
(Wiegers, Bender)

11. 9
| © 2014, Cognizant16
Boundary Ambiguity
A condition when the author uses terms - among or up to. The
scope of the requirement is ambiguous because the stated
requirement can be interpreted in multiple ways.
Example:
1. If an employee makes less than $20,000 per year, the
employer pays 100% of the healthcare premium.
2. If an employee makes more than $20,000 per year, the
employer pays 50% of the healthcare premium for the
employee.
Ambiguity: What if an employee makes exactly $20,000?
(Wiegers, Bender)
| © 2014, Cognizant17
Built-In Assumptions
A condition when the author assumes that all consumers of the
document will have the same level of domain knowledge
• Industry domain knowledge
• Subject domain knowledge
• Functional knowledge
• Environmental knowledge
Example: The system must apply the same limitations to
searches for existing groups as currently exists in Google Search.
Ambiguity: The requirement assumes the reader knows how the
functionality exists today.
(Wiegers, Bender)

12. 10
| © 2014, Cognizant18
Dangling Else
A condition when a requirement states expected results (what
normally happens) but does not state exceptions and error conditions.
Dangling Else
Can
Shall
Could
Should
Is one of
Will
Must
Would
Example: The employee address type shall be either house,
apartment or condominium.
Ambiguity: The requirement does not consider exception conditions
such as PO Box.
(Wiegers, Bender)
| © 2014, Cognizant19
Etc.
Etcetera is not a quantifiable measurement that can be confirmed so
it is considered totally ambiguous. (Phrases or sentences ending with
etc )
Example: Subscribers shall identify themselves with unique
information (policy number, social security, etc.) when they call
Customer Care for information about their policy.
(Wiegers, Bender)

13. 11
| © 2014, Cognizant20
Ambiguity Review Types
| © 2014, Cognizant21
Categorizing Ambiguities to Support Metrics
Ambiguity Categorizations will help with
• Tracking patterns of ambiguities
− For a Business Analyst
− For the Program
• Building Elicitation Checklists
− Lessons learned turned into questionnaires
− Mentoring sessions for the BA Team
• Team Performance Measurements
− Scope Management
− Defect Leakage to Production
− Requirement Defects
Purpose of Ambiguity Types

14. 12
| © 2014, Cognizant22
Ambiguity Type Description
Ambiguous Term
Terms (Phrase or Word) used in requirements which can be interpreted
by the reader in multiple ways e.g. frequently, occasionally, efficiently
Conflicting Requirement
Requirements which contradict each other – either in the same
document or across multiple documents.
e.g. The field name is Effective Date but the data type is defined as an
integer
Glossary
Word or acronym used in requirements that is new or not commonly
used but has not been defined in the Glossary/Definitions section.
Grammar, Spelling
& Wording
Grammar, spelling corrections and proposed wording improvements to
increase clarity of the requirement
Incomplete Requirement
Incomplete requirement or statement describing conditions when
information is not fully detailed preventing design or test validation
e.g. The system shall handle 15-25% increase in the second year
Missing Requirement
Missing requirements that were not documented or may not have been
elicited from the business user.
e.g. Missing requirements – alternative flows, business rules,
exceptions and error conditions (Questions – What, When, Where)
Unclear Requirement
Requirements or statements requiring further clarification to allow the
reader user to fully understand the requirement (Questions – How,
Why, What do you mean by )
(Wiegers, Bender)
| © 2014, Cognizant23
Ambiguity Review Template

15. 13
| © 2014, Cognizant24
Ambiguity Review Template
• Transfer requirements document to Excel template
− Revision History
− Introductory information (free text)
− Business Requirements
− Data presented in tables such as Glossary, Field Elements
and Financial Information segregated to its own worksheet
• Ambiguity Columns to the Right
− Ambiguity Type
− Ambiguity Description
Transform the Requirements Document to Excel
| © 2014, Cognizant25
Requirements Worksheet Example
Not all columns need to be transferred to the Ambiguity Template

16. 14
| © 2014, Cognizant26
Glossary Worksheet Example
Text transferred to Excel template
| © 2014, Cognizant27
Ambiguity Review – Keyword Searches

17. 15
| © 2014, Cognizant28
Ambiguity Review Keywords
An initial review for keywords helps to identify Incomplete
Requirements
All files are transmitted daily.
All 820 Payments Files shall be transmitted to the Federal and
Maryland Jurisdiction after the nightly batch jobs complete
• The Federal 820 Payment File shall be sent daily during the
transmission window between 11 PM and 4 AM EST
• The Maryland 820 Payment File shall be sent daily during the
transmission window between Midnight and 4 AM EST
• If no payments are made for a jurisdiction, an empty file shall
be sent to the jurisdiction
Clues to Finding Ambiguities
| © 2014, Cognizant29
Ambiguity of
Reference
Ambiguous
Adjectives
Ambiguous Adverbs Ambiguous Variables
Ambiguous
Verbs
above ordinary infrequently the database derive
below rare intuitively the field determine
it same just about the file edit
such seamless more often than not the frame enable
the previous several more or less the information improve
them similar mostly the message indicate
these some nearly the module manipulate
they standard normally the page match
this the complete not quite the rule maximize
those the entire often the screen may
Ambiguous Adjectives transparent on the odd occasion the status might
all typical ordinarily the system minimize
any usual rarely the table modify
appropriate valid roughly the value optimize
custom Ambiguous Adverbs seamlessly the window perform
efficient accordingly seldom Ambiguous Verbs process
every almost similarly adjust produce
few approximately sometime alter provide
frequent by and large somewhat amend support
improved commonly transparently calculate update
infrequent customarily typically change validate
intuitive efficiently usually compare verify
invalid frequently Ambiguous Variables compute
many generally the application convert
most hardly ever the component create
normal in general the data customize
(Bender/Wiegers)

18. 16
| © 2014, Cognizant30
Ambiguity Review – Content Review
| © 2014, Cognizant31
Ambiguity Review for Content
• Ambiguities in content focus on the following
− Conflicting Requirements
− Incomplete Requirements
− Missing Requirements
− Unclear Requirements
• Questions of Who, What, When, Where & Why
• Questions of How tend toward design questions
− Only use to clarify expected behavior or outcome
− Do not use to ask how the system will process the function
behind the scenes
Clues to Finding Ambiguities

19. 17
| © 2014, Cognizant32
Sample Ambiguity Questions
• What is the name of the file?
• Who is the user?
• What permissions does the user need to review the file?
• When is the file sent?
• What are the contents of the file?
• Where is the file sent?
• What are the business rules to validate the file?
• What are the error messages displayed when a business rule
validation fails?
• What happens if the file is corrupted and can’t be read?
• What happens if the file fails to be generated?
• Is an alert sent if a file is late, corrupt or fails?
• Who is the alert sent to?
Who, What, When, Where, Why
| © 2014, Cognizant33
Ambiguity Review Example
If the policy is a subscriber + spouse/domestic partner +
dependents policy, subscriber wants to cancel from the policy,
Exchange will send us 834 file to enroll the spouse/ domestic
partner as the subscriber under a new SID and the dependents
will remain as dependents under the new subscriber.
Note: Subscriber + Family
Content Questions Raised for this Requirement

20. 18
| © 2014, Cognizant34
Ambiguity Questions Posed
| © 2014, Cognizant35
Ambiguity Review Metrics

21. 19
| © 2014, Cognizant36
Ambiguity Review Metrics
Patterns within the Requirements Document
3
6
Top 10 Ambiguities by Type Total
Ambiguous Term 55
Grammar Spelling & Wording 79
Incomplete Requirements 59
Unclear Requirements 57
Total Ambiguities 250
| © 2014, Cognizant37
Program Level Ambiguity Review Metrics
Patterns across the Requirements Document

22. 20
| © 2014, Cognizant38
Program Level Ambiguity Review Metrics
Patterns across the Requirements Document
| © 2014, Cognizant39
Program Level Ambiguity Review Metrics
Patterns across the Requirements Document
Ambiguity Review Summary Totals
Total Requirement Considered* 3,737
Total Ambiguous Observations 26,478
Top 10 Ambiguities 7,463
Percent Ambiguities in Top Ten 28%
Average Ambiguities Per Requirement 7
*Requirements include Non-Requirement Sections such as Introduction, Purpose, etc.

23. 21
| © 2014, Cognizant40
Ambiguity Review – Benefit to QA
| © 2014, Cognizant41
QA Benefits from Ambiguity Review
• Geographically dispersed teams benefit from the additional
level of detail documented in the requirements
• Test Case Creation is simplified by clear requirements
− Normal Flow & Alternative Flows detailed in Use Cases
− Error conditions with error messages
− Business rules and constraints
− Data requirements
− Expected Outcome
• Traceability to requirements provides QA Engineer with path
to use cases and functional/non functional requirements
− Diagnosis of defect types clear
• Requirements vs. Code vs. Test Case
Quantifiable Impact

24. 22
| © 2014, Cognizant42
Ambiguity Review – Be Nice About It!
| © 2014, Cognizant43
Business Analysts Have Feelings
Ambiguity Reviews are meant to be objective and focus on the
requirements content. However You are critiquing a
Business Analyst’s work.
• Recognize the Business Analyst’s work effort
• Recognize the Business Analyst’s challenges with eliciting
requirements from business users.
• Be courteous when providing feedback.
Write Ambiguity Questions that are quantitative, clear, correct
and complete!
Be respectful of your feedback

25. 23
| © 2014, Cognizant44
Appendix
| © 2014, Cognizant45
Requirements Review Phase

26. 24
| © 2014, Cognizant46
References
Writing High Quality Requirements
By Karl E. Wiegers, 2006
The Ambiguity Review Process
By Richard Bender
Assessing the Impact of Poor Requirements on Companies
By Keith Ellis
Executive Guide to Evaluating Business Requirements Quality
By Keith Ellis
Getting Consensus on Business Requirements – Tips and Traps
By Keith Ellis
The Quest for Good Requirements
By Dr. Martin Schedlbauer, 2011
How to Prevent the Negative Impacts of Poor Requirements
By Sergey Korban, April 30, 2013
The Business Value of Better Requirements
By Karl E. Wiegers, 2006