This document discusses how to establish a clear baseline for testing user stories. It defines a baseline as a cartesian product of "what to test" and "test for what". What to test includes individual user stories and collections of user stories spanning epics. Test for what refers to acceptance criteria such as functionality, performance, security, and usability. Different types of tests are mapped to these criteria. Together, what to test and test for what form the baseline, and applying strategies like thinking and proving or executing and evaluating tests establishes a clear approach to validating user stories.

1. How to test an user story
Setting a clear baseline
Powered by HBT
T Ashok
Founder & CEO
STAG Software Private Limited
Architect - HBT
in.linkedin.com/in/AshokSTAG
Ash_Thiru
Webinar: Mar 12, 2015, 1100-1200 IST
This is the second webinar in the series on “How to test an user story”.
The focus in this webinar is on setting a clear baseline to come up with an effective approach to
validation.

2. © 2015 STAG Software Private Limited. All rights reserved. 2
An user story is seen as a modern way of communicating end user's needs &
expectations in a sweet & simple format that can be easily modified.
This brevity/simplicity hides information leading to understanding in the
small and potentially missing the big picture.
The previous webinar titled "Question to understand" focused on
how-to-identify these "white spaces" in an user story to uncover
potential gaps and understand what it is supposed to accomplish.
This webinar, the second in this series,
will focus on establishing a clear goal of
‘what-to-test-for’ and ‘how-to-test’ to
formulate an effective strategy.
Setting the context from the last webinar, the user story is seen as a modern way of communicating
the end user needs and expectations, that can be refined/modified based the user feedback post
release. It is interesting to note that the brevity and simplicity of an user story hides information
leading to understanding in the small and potentially missing the big picture.
The first webinar in this series focused on how to identify these white spaces in an user story to
uncover the potential gaps and understand what is supposed to be accomplished. In this webinar the
focus is on establishing the clear goal of “What to Test” and “How to Test” to formulate an effective
strategy.

3. © 2015 STAG Software Private Limited. All rights reserved. 3
User Story
Who
User/Persona
What
Collaborations
When
Pre-conditions
Is it?
Acceptance criteria
How much
Usage profile
Who is this meant for?
The background of user & usage
Volume, frequency concurrency
Perception of importance
... with other user stories
... other systems
System states, prerequisites
... functional : conditions of satisfaction
... non-functional attributes too
Why
Issue/benefit
How
Behaviour conditions
Implementation
Where
Environment
What are we solving?
What is the expected benefit to the user?
Understand business logic,
& implementation details
... user’s situation/ constraints
... deployment environment/ constraints
“As a <specific user/persona/role>” I want <desired feature/issue that needs to be solved>, so that <benefit from the feature>”
+ Acceptance Criteria
User story (Revisit from last webinar)
An user story is “What do I as a specific user(s)/role/persona want as features, their intended benefit
and the criteria that they satisfy. The first webinar focused on questioning commencing with
Who&Why and then moving onto How, How Much, Where, Is It & When, to enable understanding the
user story well. Remember the two core concepts of HBT “Landscaping” & “Viewpoints” that enables
one to come up with good questions on various information elements and their connections and the
perspectives on criteria, usage, perceived importance by different types of end users.
In this webinar, the focus is on understanding the collaborations between user story and other
external systems and “Is it” , the acceptance criteria. Note that the acceptance criteria is not limited to
the functionality alone and may include attributes (e.g performance, security, usability) as well.
In short, to be clear on “What do you want to test?”, “What do you want to test for?’” and ‘How to
test?’.

4. © 2015 STAG Software Private Limited. All rights reserved.
What is a “Baseline”?
4
In HBT, Baseline is a cartesian product of
What-to-test and Test-for-What
What-to-test
x
Test-for-What
Acceptance criteria
(Test types)
User story?
Collections of user stories?
So what is a baseline? In HBT, a baseline is defined as a cartesian product of “What-to-Test” and
“Test-for-What”. Let us understand “What-to-test” first. In real life usage, a typical user uses a
combination of user stories to get his/her job done. Hence what we need to test is not just a
standalone user story, but also collections of these that form a bigger flow. i.e “what-to-test” is an
individual user story and collection of user stories. As we move across sprints the combinations of
user stories become bigger mimicking real end user operations.
Now let’s understand on “Test-for-what”. In Agile Methodology, this is called ‘Acceptance Criteria’, a
set of criteria that an user story is expected to satsify if it has deliver value to the user. And to validate
these criteria, we might perform different types of test. Acceptance criteria is not limited to
functionality alone, it may include attributes requiring multiple types of tests to be performed.
Connecting these two, a baseline is about clearly understanding the user stories(& combinations) and
various types of tests to done (to meet the various acceptance criteria).

5. © 2015 STAG Software Private Limited. All rights reserved.
And Strategy is ...
5
What-to-test
x
Test-for-What
Acceptance criteria
(Test types)
User story?
Collections of user stories?
+
W
H
A
T
Baseline
How-to-do
Think & Prove
Execute & Evaluate
H
OW
Approach
Now having set a baseline, we need an approach to accomplishing this. Adding the ‘How’ to
‘What’ (the Baseline), we will have a strategy to evaluating the various user stories done in a sprint.
Having identified the various type of tests to be done, the next natural question is “How-to-do”. Can
we only execute the test and evaluate the correctness of implementation of the user story
(‘Execute&Evaluate’) or we statically evaluate by thinking through the scenarios and then proving?
(Think&Prove’)?

6. © 2015 STAG Software Private Limited. All rights reserved.
What to test ... (1/3)
6
User Activities
(Theme?)
User Tasks
(Epic?)
User Story
From http://winnipegagilist.blogspot.in/2012/03/how-to-create-user-story-map.html
Here is a nice picture of “User stories of an email client” from the blog mentioned. The orange cards
at the top represent the the highest level of activities that user performs using the email service –
Organize Email, Manage Email, Mange Calendar, Manger Contacts which we can refer to as Theme.
At next level it is broken down further as a ‘big user story’– Search Email, File Emails, Compose Email,
Read Email, Delete, View Calendar etc.. , which we refer as Epic.
The ’Search email epic’ is made of combination of these user stories - ‘Search by keyword’ , ‘Limit the
search to one field only’, ‘Search limit of more than one fields’, ‘Search attachments’, ‘Search sub
folders’, which are implemented in multiple sprints/releases..

7. © 2015 STAG Software Private Limited. All rights reserved.
What to test ... (2/3)
7
User Activities
(Theme?)
User Tasks
(Epic?)
User Story
From http://winnipegagilist.blogspot.in/2012/03/how-to-create-user-story-map.html
2 : Test a set of user stories of an epic
1 : Test an user story
3 : Test a set of user stories that an
user would do in a sequence (flow)
So what do we test?
Test an individual user story ‘Search emails by keyword’
We can test combinations of user stories that form an epic
Test a set of user stories of an epic - ‘Move emails into sub-folders’
(3)We can test sequence of user stories across epics that form a flow that are implemented in a
sprint/release - Create an appointment, update the location and then view the same
and lastly
(to next slide)

8. © 2015 STAG Software Private Limited. All rights reserved.
What to test ... (3/3)
8
User Activities
(Theme?)
User Tasks
(Epic?)
User Story
From http://winnipegagilist.blogspot.in/2012/03/how-to-create-user-story-map.html
2 : Test a set of user stories of a epic
1 : Test an user story
3 : Test a set of user stories that an
user would do in sequence (flow)
4 : Test a set of user stories across
releases(sprints) & epics
(4) Test a set of user stories across sprints(releases) and epics - Create contract, update the contact
info, then Add address data, update this and then delete this.
This represents a typical sequence of real life usage.

9. © 2015 STAG Software Private Limited. All rights reserved.
Real life usage (remember from last webinar...)
9
1
4
7
2
5
8
3
6
9
Other
System1
Other
System2
... is a collection of user
stories strung together.
Ultimately we need to
validate the various flows.
Hence what-to-test is a user story initially and combinations of user stories spanning across epics
representing user flows. Real life usage is collection of user stories. As we progress across sprints/
releases, we validate longer flows from the end user view point, with other system(s) integrated.

10. © 2015 STAG Software Private Limited. All rights reserved.
Test for what - CRITERIA
10
CC1 CC2 CC3 CC4 ... CCn
E1
✓ ?
E1 -------
-------
E2
✓ ✓ ✓
E2 -------
-------
-------
-------
-------
-------
E3
✓
E3 -------
-------
E4
✓ ✓ ✓
E4 -------
-------
-------
-------
-------
-------
E1 : An user story E2 : Set of user stories of a epic
E3 : Set of user stories used in a sequence (flow)
E4 :Set of user stories across releases(sprints) & epics
Cleanliness CriteriaCleanliness Criteria
CC1 Functionality
CC2 Capacity/Load
CC3 Performance
CC4 Security
CC5 Usability
... ...
Acceptance Criteria
... Instantiation of CC
... Satisfaction of conditions
Attribute condition(s)
t<=2s
Behaviour conditions
Combination of conditions
Recap that baseline is the combination of two parts
1. What to Test
2. Test for What
‘Test for What’ are the criteria that we need to consider to meet the expectations- the acceptance
criteria. The entity to test as we understood are E1-Individual User Story, E2- Set of User stories of an
epic, E3- Set of user stories used in a sequence or the flow and E4 – Set of user stories across
releases & epics.
An entity under test can be listed against the acceptance criteria so that it should not miss the
acceptance criteria which are given to us. But we have to make sure that the acceptance criteria is
indeed complete and also not ambiguous.
In HBT methodology, expectations (of quality) is stated in terms of Cleanliness Criteria for e.g.
Functionality, Capacity, Performance, Security, and Usability. So setting up ‘test for what’ is listing
down these cleanliness criteria and mapping against the entities. Functionality will be applicable
across all, while others like Performance may be applicable for some. We have to qualify these
criteria and hence these have to be unambiguous. So in order to test each of these elements of what
to test, we will consider the list of cleanliness criteria, choose these applicable and elaborate. This
approach will enable us to discover if stated acceptance criteria is indeed complete.

11. © 2015 STAG Software Private Limited. All rights reserved.
Test for what - TEST TYPES i.e. PDTs*
11
TT1 TT2 TT3 TT4 ... TTn
E1
✓ ?
E1 -------
-------
E2
✓ ✓ ✓
E2 -------
-------
-------
-------
-------
-------
E3
✓
E3 -------
-------
E4
✓ ✓ ✓
E4 -------
-------
-------
-------
-------
-------
E1 : An user story E2 : Set of user stories of a epic
E3 : Set of user stories used in a sequence (flow)
E4 :Set of user stories across releases(sprints) & epics
Cleanliness CriteriaCleanliness Criteria TT
CC1 Functionality FT
CC2 Capacity/Load LT
CC3 Performance PT
CC4 Security ST
CC5 Usability UT
... ...
Acceptance Criteria
... Instantiation of CC
... Satisfaction of conditions
Attribute condition(s)
t<=2s
Behaviour conditions
Combination of conditions
Regression
*PDT = Potential Defect Type
So if these Cleanliness criteria have to satisfy each of the entity, this means that the ‘Potential Defect
Type’ that could affect the functionality or security should be uncovered. This means we will have to
perform specific type of test(s) like Performance, Functionality etc as applicable. The Cleanliness
criteria leads to the Potential Defect Type that is closely associated with the type of test.
When we are applying an agile methodology we are constantly modifying, adjusting and evolving.
Hence as we go across the sprints changes could be done in terms of design or code to any user
story and hence regression is required. Regression is not limited to the functional tests only, note that
it is necessary to validate any of the acceptance criteria are not violated when changes are done.

12. © 2015 STAG Software Private Limited. All rights reserved.
Quality Levels & Test Types
12
L9 End user value End user value testEnd user value test
L8 Deployment correctness Installation testInstallation test Migration testMigration test
L7 Attributes correctness LSPS test Reliability testReliability test Security test
L6 Environment cleanliness Good citizen testGood citizen test Compatibility testCompatibility test
L5 Flow correctness Flow test
L4 Behaviour correctness Functionality testFunctionality test Access control testAccess control test
L3 Structural cleanliness Structural testStructural test
L2 Interface cleanliness API validation testAPI validation test GUI validation testGUI validation test
L1 Input cleanliness Input validation testInput validation test
Naturalorder(Qualitygrowth)
TT1 TT2 TT3 TT4 ... TTn
E1 ✓ ?
Natural order (Quality growth)
Note that natural
order of tests and
therefore the growth
of quality.
Note we can scope
the test to be
conformance,
robustness,
or both based on the
sprint objective.
The various types of test to be done can be arranged in a very natural order, depicting progression
in terms of quality growth. This results in a layered mode of validation : Starting from L1 (Level #1)
where we check that bad input(s) is/are rejected to L2 where interface that accepts inputs is clean to
L3 where structural aspects of the entity are validated. These done, we are ready to validate the
functional behaviour of the individual user story (L4) moving onto validating behaviour of user story
collection(L5). Moving to L6, to ensure that flow do-not corrupt the environment and are not affected
by the environment and then later validating the various attributes of the flows at L7 and finally
validating with real deployment environment. at L8. Note that tests are to conducted in these natural
order in a given sprint to be efficient.

13. © 2015 STAG Software Private Limited. All rights reserved.
Now where are we? [And Strategy is ...]
13
What-to-test
x
Test-for-What
Acceptance criteria
(Test types)
User story?
Collections of user stories?
+
W
H
A
T
Baseline
How-to-do
Think & Prove
Execute & Evaluate
H
OW
Approach
✓
Strategy is Baseline + Approach the WHAT and the HOW.
Arranging tests cases and executing tests in this natural order allows us to be effective and efficient .
Should we execute the tests only by dynamic execution of test cases ? or Can we use intellect to
“Think & Prove”?

14. © 2015 STAG Software Private Limited. All rights reserved.
Now where are we? [And Strategy is ...]
14
How-to-do
Think & Prove
Execute & Evaluate
H
OW
Approach
Think & Prove
Evaluating statically, very applicable to L1-L3 quality levels.
Read in Frictionless Development Testing
Execute & Evaluate
Dynamic evaluation. Design test cases & execute (Manual/
Automated)
Approach or the Strategy means “how-to validate that acceptance criteria listed for each entity under
test is met”.
This can be done is two ways
Static Evaluation : Evaluating meeting/violating acceptance criteria not by dynamic execution by static
proving using intellect. This is especially useful in Levels 1 through 3. “Think & Prove”
Dynamic Evaluation: Executing test cases for a test manually or using a tool. “Execute & Evaluate”

15. © 2015 STAG Software Private Limited. All rights reserved.
Clear baseline in short:
15
TT1TT1TT1 TT2TT2TT2 TTn
Ex
✓
T&P
E&E
Regress
✓
T&P
E&E
Regress
... TTn
Ex
✓
T&P
E&E
Regress
✓
T&P
E&E +ve | -ve
... TTn
Ex
✓
T&P
E&E
+ve | -ve
✓
T&P
E&E +ve | -ve
... TTn
Ex
-------
-------
-------
-------
-------
-------
-------
-------
-------
-------
-------
-------
E1 : An user story E2 : Set of user stories of a epic
E3 : Set of user stories used in a sequence (flow)
E4 :Set of user stories across releases(sprints) & epics
Acceptance Criteria
... Instantiation of CC
... Satisfaction of conditions
Test types to be doneApproach & Scope
1 2
4 3
So what is a clear baseline?
1. Knowing clearly as to what the element under test is in a sprint
2. Knowing clearly as to what the acceptance criteria, which is really a instantiation of cleanliness
criteria that is unambiguous and includes attributes. To ensure no ambiguity, enumerate each
criteria as as set of conditions.
3. The various types of tests to be done to validate that the criteria is met.
4. And finally the “Approach & Scope” i.e. based on the sprint scope, do we test only conformance or
both conformance &robustness and do we regress?

16. © 2015 STAG Software Private Limited. All rights reserved.
Hypothesis Based Testing - HBT
16
System Under
Test
Cleanliness
Criteria
Potential Defect
Types
Test CasesRequirements
traceability
“what to test”
Fault
traceability
“test for what”
should satisfy impeded by
Click here to know more about HBT.
http://stagsoftware.com/blog?p=570
HBT or Hypothesis Based Testing is a scientific personal test methodology where we hypothesize
potential defect types to be uncovered to meet the expectations (Cleanliness Criteria) of the needs
(System under test) via the Test cases.

17. © 2015 STAG Software Private Limited. All rights reserved. www.stagsoftware.com
HBT is the intellectual property of STAG Software Private Limited.
STEMTM is the trademark of STAG Software Private Limited.
@stagsoft
blog.stagsoftware.com
Connect with us...
Thank you. Powered by HBT
How to test an user story : Setting a clear baseline