The document discusses the shortcomings of requirements-based testing in software development, highlighting the importance of distinguishing between real business requirements and product/system specifications. It emphasizes that reliance on user stories and acceptance criteria alone may overlook critical requirements, leading to ineffective testing outcomes. The webinar aims to equip participants with the ability to evaluate test design techniques and the effectiveness of testing against true business needs.

1. XBOSoft, Inc. All Rights Reserved. 1
Not Your Grandfather’s
Requirements-Based Testing:
Do Agile User Stories or ATDD Fix It?
with Robin Goldsmith
#TestCaseTips

2. XBOSoft
Dedicated to Software Quality Improvement
Founded in 2006
We speed products to market with our expert:
• Software QA consulting
• Software testing
Global team with offices in San Francisco & Beijing
“Thorough, accurate and fast”
XBOSoft, Inc. All Rights Reserved. 2

3. House Rules
 Participants other than the speakers are muted
 Ask questions in the GoToWebinar control on the
right side of your screen or through Twitter
@XBOSoft
 Questions may be asked throughout the webinar -
we’ll try to answer them at the end
 You’ll receive info on recording after the webinar
Webinar Hashtag: #TestCaseTips
XBOSoft, Inc. All Rights Reserved. 3

4. Meet Our Speakers
Philip Lew
CEO and Founder, XBOSoft
• Relevant specialties and
passions
o Software quality process,
evaluation, measurement
and improvement
o Software quality in use /
UX design
o Mobile User Experience
and usability
o Cycling and travel
XBOSoft, Inc. All Rights Reserved. 4
Rachel Geibig
Marketing Manager, XBOSoft
• Emails you about our
industry hot topic
webinars
• Invites you to download
our latest trends in
software testing white
papers
XBOSoft Team

5. Robin Goldsmith
XBOSoft, Inc. All Rights Reserved. 5
Author of:
• ArtechHouse book Discovering REAL
Business Requirements for Software
Project Success
• Forthcoming book Cut Creep—Put
Business Back in Business Analysis to
Discover REAL Business Requirements for
Agile, ATDD, and Other Projects
• Numerous articles and featured
presentations
• The Proactive Testing™, Proactive SQA™,
REAL ROI™, and Beyond the Textbook
Software Acquisition™ methodologies
• Subject expert on requirements and testing for TechTarget’s
SearchSoftwareQuality.com, and subject expert/reviewer for the IIBA Business
Analysis Body of Knowledge (BABOK v2)
• Works directly with and trains business/systems professionals in quality and testing,
requirements analysis, software acquisition, project management and leadership,
metrics, ROI, and process improvement
www.gopromanagement.com
robin@gopromanagement.com

6. http://psqtconference.com/
Keynote and Featured Presentations
by World Renowned Quality Leaders and Experts
Ed Adams
CEO, Security
Innova
Jennifer Bonine
VP of global
delivery and
solutions for
tap|QA
Isabel Evans
Independent
Consultant
Philip Lew
CEO, XBOSoft
Robin Goldsmith
International
Institute for
Software Testing

7. XBOSoft, Inc. All Rights Reserved. 7
Bhushan Gupta,
Gupta Consulting, LLC.
Robin Goldsmith,
Go Pro Management, Inc.
Johanna Rothman,
Rothman Consulting Group, Inc.
Peter Khoury,
Magnetic Speaking
Rex Black,
RBCS
Darlene Greene,
Consultant
PNSQC 2016 Keynote and Invited Speakers www.PNS.org

8. 1. Take your
requirements
2. Test that each
one has been
satisfied
correctly
3. Duh!
“In grandfather’s day, requirements-
based tests were no big deal.”

9. Why We’re Here
XBOSoft, Inc. All Rights Reserved. 9
Requirements-based tests
• Are fundamental
• Seem obvious
Except
1. What requirements?
2. What are requirements?
3. How to identify tests based on requirements?

10. Objectives
You should be able to:
• Recognize various test design techniques that
some equate with requirements-based testing
• Describe the significance of testing against REAL
business requirements vs. against
product/system/software requirements
• Critically evaluate effectiveness of tests based on
user stories and their acceptance criteria

11. Is one of your issues testing that
requirements are met?
Awareness is a start.
XBOSoft, Inc. All Rights Reserved. 11

12. Agenda
XBOSoft, Inc. All Rights Reserved. 12
• Low-level test design techniques that some
claim = requirements-based testing
• REAL business requirements distinctions and
significance for testing
• Challenges identifying the requirements
• Strengths and issues of Agile user stories and
their acceptance tests

13. What Does “Requirements-Based
Testing” Mean to You?
Check all that apply
1. Confirming documented requirements have been met
2. Confirming undocumented requirements are met
3. Relying on users to do the testing
4. Using decision tables, decision trees, and/or cause-
and-effect graphing to identify business rule
conditions
5. Exercising use cases
6. Using some other specific test case design technique
7. Other

14. Pitfall:
Gurus Equate RBT with Particular
(Low-Level) Techniques
• Appropriating the term, typically implying it’s only
one or a few specific structured methods to
design test cases by systematically elaborating
each business rule, e.g.
• Decision trees and tables
• Logic mapping (including Cause and Effect
Graphing), like white box but of functionality
XBOSoft, Inc. All Rights Reserved. 14

15. Pitfall (continued)
• Exercising use cases, albeit systematically
• Focusing only on such methods can be forest-
and-trees
• Can mask other, especially bigger, missed
requirements
• Large number of tests identified can be
overwhelming
• Yet, still can overlook significant
requirements to test
XBOSoft, Inc. All Rights Reserved. 15

16. Some Equate RBT to
Decision Trees
Find by ID Order
Exit
Find by Phone No.
Search by Name
Search Name in Zip
Add Vendor

17. More Equate to Decision Tables
Especially helpful for business rules.
Is this correct?
Conditions
Actions
Test Cases
Scenarios
ID match Y N
Phone match Y N
Name select Y N
NameZip select Y N
Add Vendor Y
Order Y Y Y Y Y

18. Decision Tables Issues
• Still easy to miss conditions
• Does not reflect sequence, or
• Helps with combinations of conditions, to a point
Conditions
Actions
Test Cases
Scenarios
ID match Y N N N N N
Phone match Y N Y N N N N N N
Name select Y N Y Y N N N
NameZip select Y N Y Y Y N
Add Vendor Y Y Y
Exit Y
Order Y Y Y Y Y Y Y Y Y Y Y Y Y
Exit Y

19. Degrees of Structural Coverage
1. Execute module/routine from entry to exit
Within a module/routine, execute all:
2. Lines of code/steps
3. Branches (basis paths, complete)
4. Logic paths (exhaustive)
• Flowgraph: Node
 Module’s Entry, Exit
 Where logic branches
 Junctions (logic returns)
• Flowgraph: Edge
 all logic between nodes
IF
IF
T F
T
F
“Flowgraph”

20. Others Equate to Cause-Effect
Graphing
Paid
Grace
At
fault
Uninsured
Pay_
Claim
or
and
~ not
Pay the damage claim when
the insured is at fault or the
accident is caused by an
uninsured motorist and their
premium has been paid, or
when they are within the
payment grace period.Tool uses structural paths approach to
optimize rigorous tests of all variants of
multiple sequenced decisions

21. Still More Equate to Executing
Use Case Paths/Scenarios
Defined as “how an actor interacts with the system.”
The actor is usually the user, and the system is what the
developers expect to be programmed. Therefore, use cases
really are white box/design rather than black box/business
requirements. Flowgraph this Use Case. Path=Test Case
U1.Enter Vendor ID R1.1 Vendor is found (U4)
R1.2 Vendor is not found (U2)
U2.Enter vendor name R2.1Select vendor from list (U4)
R2.2Vendor is not in list (U3)
U3.Add vendor R3 Vendor is added (U4)
U4.Enter order R4 Order is entered (Exit)

22. Flowgraph of Use Case
U1
R2.2
Exit
U3
R3
R1.2R1.1
U4
R2.1
U2
R4
• U1-R1.1-U4-R4-Exit
• U1-R1.2-U2-R2.1-U4-R4-Exit
• U1-R1.2-U2-R2.2-U3-R3-U4-
R4-Exit
• U1-R1.2-U3-R3-U4-R4-Exit
• U1-R1.2-Exit
• U1-R1.2-U2-R2.2-Exit
• U1-R1.2-U2-R2.2-U3-Exit
• U3-R3.1-Exit
• U4-R4.1-Exit
R3.1
R4.1

23. Equating to Any Specific (Typically
Business Rule-Oriented) Technique
• Can’t be right—excludes requirements and/or tests
identified by other claimed techniques
• Does not address other types of requirements
– Functionality
– Calculations, formulas, algorithms
– Data variants and database structure
– Quality factor “-ilities,” e.g., usability, security,
performance, reliability, maintainability, etc.
• Irrelevant for overlooked requirements
XBOSoft, Inc. All Rights Reserved. 23

24. Pitfall:
RBTs Are Only as Good as the
Requirements
• Most organizations’ aren’t very good – unclear, wrong,
incomplete – and are actually even worse than realized
because most miss REAL requirements
• The REAL requirements aren’t what you happened to write
down, they’re what you ought to have known about and
captured so they aren’t forgotten, including things like
quality factors and performance levels.
• Most of what people call “requirements” actually are design.
Design doesn’t have to be engineering or technical. A
product or system one intends to create is a design, albeit
high-level.

25. Error Sources by Phase
Portion of
System Correct
Portion of
System In ErrorPhase
Requirement missed,
extra, or misdefined
Requirements
Required and defined appropriately
Included in design and designed
appropriately
Design
Included in product/system and
built appropriately
Build & Test
Included in finished system
and installed appropriately
Implementation
Missing from design
or inappropriate
Missing or built
inappropriately
Missing from or
inappropriate in finished system

26. IT (and Other?) Project Economics
• Maintenance is 66-90% of system cost
• Maintenance is mainly completion/ correction of
development (wrong/missed requirements)
• 2/3 of finished system errors are requirements
and design errors
• Fixing a requirements error will cost
• 10X+ during development/construction
• 75-1000X+ after installation (maintenance)

27. Do your organization’s routine
measures show these effects?
What’s the relevance to
requirements-based tests?
XBOSoft, Inc. All Rights Reserved. 27

28. Two Types of Requirements:
Business/User/Customer Product/System/Software
• Business/user/stakeholder/
customer language & view,
conceptual; exist within the
business environment
• Serves business objectives
• What business results must
be delivered to solve a
business need (problem,
opportunity, or challenge)
and provide value when
delivered/satisfied/met
• Language & view of a human-
defined product/system
• One of the possible ways: How
(design) presumably to
accomplish the presumed
business requirements
• Often phrased in terms of
features/external functions each
piece of the product/system
must perform to work as
designed (Non-/Functional
Specifications)Many possible ways to accomplish

29. Even Requirements “Experts” Think
the Difference Is Just Level of Detail
Business Requirements
(High-Level, Vague)
Product/
System/
Software
Reqs.
(Detailed)
BABOK v3 2.3 p. 26
“Business requirements: statements
of goals, objectives, and outcomes
that describe why a change has been
initiated.”
Mistaken “Levels Model”

30. When Business/User Requirements
Are Detailed First, Creep Is Reduced
Business Requirements
(High-Level)
Business
Product/System/Software
Reqs.(High-Level)
Reqs.
(Detailed)
Reqs.
(Detailed)
Product/
System/
Software

31. Requirements Overview
Stakeholders
Business needs,
problems, value
Discovery
Analysis
High-Level & Detailed
REAL Business/
Stakeholder Requirements
Deliverable Whats  Value
Product/System
/ Software
Requirements
Features Hows
Respond to
Functional Requirements
Use Cases
Software Requirements Specifications
[Non-Functional Requirements]
Quality Factors, Attributes, ‘Ilities’
(Supplemental Specifications)
User/
(Usage)
High-Level
Detailed
Technical/
Engineering
Design
Code

32. Tests Based on Product/System/
Software Features/Requirements
• What’s usually meant by “requirements”
• Necessary but not sufficient
• Can pass even though product is not right
XBOSoft, Inc. All Rights Reserved. 32
“Conventional We’s Dumb”

33. More Effective Tests Are Based on
REAL Business Requirements
• Demonstrate that the
• Product/system/software how’s
• Satisfy the REAL business requirements
deliverable what’s
• And thereby provide REAL value by
achieving objectives
XBOSoft, Inc. All Rights Reserved. 33
“Unconventional Wisdom”

34. Pitfall:
Just User Story Requirements
and Their Acceptance Test
• Assuming all that’s needed is writing user
stories and acceptance tests for them
• User story brevity means lots of requirements content
may be taken for granted or simply overlooked
• Ditto for user story acceptance criteria/tests, easily not
as accurate or thorough as assumed
• Several common Agile dynamics can cause user story
requirements content to deteriorate
• Consider effects of discarding
• And…

35. Requirements in Agile Are
Generally Considered User Stories
As a <type of user>
I <want/can/am able to/need to/etc.>
so that <some reason>
Mike Cohn
“User Stories, Epics and Themes”
http://www.mountaingoatsoftware.com/blog/stories-epics-and-themes
Some call “placeholder for conversation”

36. User Stories Theory
Stakeholders
Business needs,
problems, value
Discovery
Analysis
High-Level & Detailed
REAL Business/
Stakeholder Requirements
Deliverable Whats  Value
Product/System
/ Software
Requirements
Features Hows
Respond to
Functional Requirements
Use Cases
Software Requirements Specifications
[Non-Functional Requirements]
Quality Factors, Attributes, ‘-ilities’
(Supplemental Specifications)
User/
(Usage)
High-Level
Detailed
Technical/
Engineering
Design
Code
User
Stories
C
O
N
V
E
R
S
A
T
I
O
N
S

37. User Stories Actually Are a Bit
More
• Card
• As a <role>
• I want <something>
• So that <benefit>
• Conversation
• Confirmation
• User story acceptance criteria, tests
What do conversations do to the user
stories requirements and their tests?
Working code

38. Too Often User Stories Practice
Stakeholders
Business needs,
problems, value
Discovery
Analysis
High-Level & Detailed
REAL Business/
Stakeholder Requirements
Deliverable Whats  Value
Product/System
/ Software
Requirements
Features Hows
Respond to
Functional Requirements
Use Cases
Software Requirements Specifications
[Non-Functional Requirements]
Quality Factors, Attributes, ‘Ilities’
(Supplemental Specifications)
User/
(Usage)
High-Level
Detailed
Technical/
Engineering
Design
Code
User
Stories
Conver
sation

39. REAL Business Requirement:
Calculate 5% Sales Tax on the Order
• Requirements-Based Test:
• Enter one item at $99.99, Sales tax should be $5.00
• Acceptance Criteria
• Create an order consisting of both taxable and
nontaxable items
• Create an order that covers more than one page
• Produce a credit voucher for a returned item
• Calculate for states and localities with different rates
Content is the important part, not format.
= or reveal requirements?

40. You Should Be Able to:
• Recognize various test design techniques that
some equate with requirements-based testing
• Describe the significance of testing against
REAL business requirements vs. against
product/system/software requirements
• Critically evaluate effectiveness of tests based
on user stories and their acceptance criteria

41. Go Pro Management, Inc. Seminars/Consulting--Relation to Life Cycle
Proactive Systems/Software Quality Assurance (SQA)™
Software, Test Process Measurement & Improvement
Feasibility
Analysis
Systems
Analysis
System
Design
Develop-
ment Implement-
ation Operations
Maintenance
Proactive Testing™:
Risk-Based Test Planning,
Design, and Management
Testing Early in the Life Cycle
Re-Engineering: Opportunities for IS
Credibly Managing Projects and Processes with Metrics
21 Ways to Review Requirements
Making You a Leader
Managing Software Acquisition and Outsourcing:
> Purchasing Software and Services
> Controlling an Existing Vendor’s Performance
Proactive User Acceptance Testing
Reusable Test Designs
Test Estimation
Risk
Analysis
Defining and Managing
Business Requirements
Writing Testable SW Requirements

42. Post your webinar questions on Twitter @XBOSoft
Registrants will receive an email with information on where to view the recording
and slides from today’s webinar.
Join us to keep updated on all our webinars, reports and white papers:
facebook.com/xbosoft
+xbosoft
linkedin.com/company/xbosoft
Check out our blog: http://xbosoft.com/software-quality-blog/
Download our free white papers: http://xbosoft.com/knowledge-center/
Email us with ideas for future webinars or questions regarding our services!
services@xbosoft.com
Thank you!
Q+A
www.xbosoft.com