Specification-By-Example
with Gherkin
Christian Hassa,TechTalk
(ch@techtalk.at, @chr99ha)

Almost 4 years later …
#106 inVisual StudioGallery (30.1.2013)
#135 in NuGet (30.1.2013)
~800 visits daily
~ 4000 active users
> 30 contributors

Goals
Make you curious about Specification-By-Example
New perspective for SpecFlow users

About me
• Managing Partner atTechTalk
• Agile consulting and delivery
• Based in Europe:
Vienna, Budapest, Zurich

Terminology
• Specification-By-Example
• AcceptanceTest Driven Development (ATDD)
• Behaviour Driven Development (BDD)

Agile requirements

Purpose of user stories?
• Describe user needs
• Unit of prioritization
• Unit of planning
• Reminder for a discussion
• Mechanism to defer detail

Requirement levels
Impact Mapping
Story Mapping
Specification-By-Example
Why?
How?
Acceptance
Criterion
Epic
Deliverable, Outcome
Impact, Goal
Easier to define upfront Harder to define upfront
Reminder
for a
discussion
Bug
report
Isolated,
formalized
example
User Activity
User Story
Example
Code

Collecting acceptance criteria
“I would try to put a book into
the shopping cart …”
“I would try to remove a book
from the shopping cart…”
“I’d check whether the shopping cart
is empty, when I enter the shop …”
“I would try to add the same book
again to the shopping cart …”
Books can be placed into shopping cart.
Books can be removed from shopping cart.
Shopping cart should be empty when
entering the shop.
Adding same book again to shopping cart
should increase quantity.
As a potential customer
I want to collect books in a shopping cart
So that I can order several books at once.
“Imagine this story is already implemented:
What would you try out?”

Using examples

Examples for user stories
UI wire frames,
existing UI
rules, key examples
existing artifacts,
mock-ups

Discussion of acceptance criteria
Original idea for the illustration:George Dinwiddie
http://blog.gdinwidiee.com
public void TestInitialOrderDiscount()
{
Customer newCustomer = new Customer();
Order newOrder = new Order(newCustomer);
newOrder.AddBook(
Catalog.Find(“ISBN-0955683610”)
);
Assert.Equals(33.75,
newOrder.Subtotal);
}
Register as “bart_bookworm”
Go to “/catalog/search”
Enter “ISBN-0955683610”
Click “Search”
Click “Add to Cart”
Click “View Cart”
Verify “Subtotal” is “$33.75”
We would like to encourage new users to buy in our shop.
Therefore we offer 10% discount for their first order.

Specification-By-Example
Examples …
– make abstract descriptions
better understandable
– are usually not formally
exchanged or documented
Brian Marick
Examples Tests
Requirements
consist of
describe verify
fulfillment of

Discussion of acceptance criteria
Original idea for the illustration:George Dinwiddie
http://blog.gdinwidiee.com
public void TestInitialOrderDiscount()
{
Customer newCustomer = new Customer();
Order newOrder = new Order(newCustomer);
newOrder.AddBook(
Catalog.Find(“ISBN-0955683610”)
);
Assert.Equals(33.75,
newOrder.Subtotal);
}
Register as “bart_bookworm”
Go to “/catalog/search”
Enter “ISBN-0955683610”
Click “Search”
Click “Add to Cart”
Click “View Cart”
Verify “Subtotal” is “$33.75”
We would like to encourage new users to buy in our shop.
Therefore we offer 10% discount for their first order.

Illustrated with shared examples
Original idea for the illustration:George Dinwiddie
http://blog.gdinwidiee.com
Given the user has not ordered yet
When the user adds a book with the price of USD 37.50 into the shopping cart
Then the shopping cart sub-total should be USD 33.75.

Discover new aspects
Original idea for the illustration:George Dinwiddie
http://blog.gdinwidiee.com
Actually, this not quite right:
Books on sale should be excluded.

Collaboration: 3 amigos
“Happy
Path”
Technical
feasability
Exceptions,
border cases

Abstract acceptance criteria
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Books can be removed from the shopping basket
Shopping basket is initially empty
The same book can be added multiple times to the shopping basket

Formalizing examples
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Given my shopping basket is empty
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 1 copy of “Harry Potter”

Formalizing examples
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basketThe same book can be added multiple times to the shopping basket
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Given my shopping basket contains 1 copy of “Harry Potter”

Example structure
The same book can be added multiple times to the shopping basket
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Given my shopping basket contains 1 copy of “Harry Potter”
Title: Describes isolated, focused intention=abstract acceptance criterion
Arrange: Context, initial state of the system
Act: Execution of the feature
Assert: Assertion of observable behavior
And I should see the warning: “Book already existed in basket”
Triple-A
constraint
“Checks”
Steps can be chained up
Evolution of a domain specific
test and specification language.

Life time of examples

Purpose of examples
• Shared understanding:
acceptance criteria
• Documentation:
system details
• Regression-tests:
violated assumptions

Continuous validation
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Given my shopping basket contains 1 copy of “Harry Potter”
„Step Definitions“ are binding individual steps
to an automatable interface of the application.
Automation does not necessarily have to bind to the UI.
Automation of system is supported/evolving with development.
SystemUI Automation

Demo
Gherkin automation for .NET
• Visual Studio plugin (VS-Gallery)
• NuGet Package
http://www.specflow.org

Impact on testing

Test automation becomes expensive, when …
• trying to automate manual tests
• making tests unreadable when
automating them
• automating only after
completing implementation
structure
readability
when to test

Structure
Manual tests
Asserts Multiple combined features
Structure ACT-ASSERT-
ACT-ASSERT-
ACT-ASSERT-
…
Dependent on existence of other features
Long test path with high chance to break
Cause and impact of an error hard to
trace.
Automated Check
Single aspect of a single feature
ARRANGE –
ACT –
ASSERT
Independent of other features
Short test path with lower chance to break
Cause and impact of an error easier to
trace.

Test automation pyramid
User
journeys
Acceptance-
criteria
Units
many
harder
easier
Automation
few
exploratory
testing
Source: Mike Cohn

Readability
// Go to web page 'http://localhost:40001/' using new browser instance
BrowserWindow localhostBrowser = BrowserWindow.Launch(
new System.Uri(this.RecordedMethod1Params.Url));
// Click 'Register found item' link
Mouse.Click(uIRegisterFoundItemHyperlink, new Point(56, 9));
// Click 'Save' button
Mouse.Click(uISaveButton, new Point(44, 14));
int foundItemNo1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9));
// Click 'Register found item' link
Mouse.Click(uIRegisterFoundItemHyperlink, new Point(63, 7));
// Click 'Save' button
Mouse.Click(uISaveButton, new Point(34, 11));
int foundItemNo2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9));
Assert.IsTrue(foundItemNo1 + 1 == foundItemNo2);
// Click 'Close' button
Mouse.Click(uICloseButton, new Point(26, 11));

A readable test case
Scenario: New found items should receive a consecutive number for the
current year
Given the previous found item of the
current year had the number 145
When I register a new found item
Then the last found item of the
current year should have the number 146

When to test
Acceptance criteria
(ATDD, BDD)
UnitTests
(TDD)
business view
technical view
Exploratory tests
Workflow tests
Performance, Scalability,
Usability,Security, …
definingtheproduct
criticizingtheproduct
New dimension: defining the product
Synergy: Specification of requirements and tests
AgileTesting Quadrants: Brian Marick

Manual testing is still necessary!
User
journeys
Acceptance-
criteria
Units
exploratory
testing
Manual Check
after story done
Main
success paths
Undiscovered
acceptance criteria
No/(few) manual
regression checks
Few paths
are enough
More time
for exploration

Cross-functional work
Sprint 1 Sprint 2 Sprint 3
short iteration
US4
Plan
Implement &
autom. test
US5
Plan
Implement &
autom. test
US2
Plan
Implement &
autom. test
US3
Plan
Implement &
autom. test
US6
Plan
Implement &
autom. test
US1
Plan
Implement &
autom. test
US7
Plan
Implement &
autom. test
US8
Plan
Implement &
autom. test
US9
Plan
Implement &
autom. test
ExploratoryTests
Specification and test
Collaboration for defining
acceptance criteria
Collaboration for
automation
Preventing bugs
instead of finding them!
Extension of
“Test Cases”
Limit
WIP
Collaboration
in manual
testing

Impact on development

TDD workflow

Extending TDD for business

Transparency for stakeholders
In Progress

Current sprint report: all scenarios

Starting with first scenario

Finishing first scenario

Progressing on scenarios

Progressing on scenarios

Progressing on scenarios

Progressing on scenarios

First user story ready for testing

Earlier start of manual testing

Done work can break again

Done work can break again

Business readable error reports

Living documentation

Higher level scenario structure
Story Mapping
Why?
How?
Acceptance
Criterion
Epic
Deliverable, Outcome
Impact, Goal
User Activity
User Story
Example
Code
Easier to define upfront Harder to define upfront

Story Maps
• Original concept by Jeff Patton
• Support iterative product design
• Optimized for desired outcome or
deliverable the system should support

Building story maps
Get popular books
delivered fast and
conveniently
Find
book
Collect
books
Order
Wait for
delivery
Receive
delivery
time
browse
best
sellers
shopping
basket
enter
address
delivery
slip
delivery
notification
pay with
credit
card
search
book by
title
wish list
inquiry
order
status
desired outcome
or deliverable
user
activities
system
features
necessity

Walking
skeleton
Enabling build–measure-learn
Get popular books
delivered fast and
conveniently
Find
book
Collect
books
Order
Wait for
delivery
Receive
delivery
time
browse
best
sellers
enter
address
delivery
slip
pay with
credit
card
search
book by
title
wish list
inquiry
order
status
necessity
shopping
basket
delivery
notification
manual
work-
around
omitted
steps
desired outcome
or deliverable
user
activities
system
features

Story Map example

Sprint 1

Sprint 2

Sprint 3

Sprint 4

Dropped Scope

Added scope

User Stories vs. Features
Product/Sprint Backlog
User Story 1
AccCrit 1
AccCrit 2
User Story 2
AccCrit 3
AccCrit 4
Living Documentation
Feature 1
AccCrit 1
AccCrit 2
Feature n
AccCrit 4
AccCrit m
User Story n
AccCrit 5
AccCrit m
AccCrit 3
AccCrit 5
„Done“
• Future options of the system
• Organized/refined according to
priority, value, effort, risk, ...
• Next possible increments of
the product (units of work)
• Current state of the system
• Organized/refined for
functional overview
• Versioned and maintained
together with source code

Living documentation
• Link business
readable
automation
from source
code to story
maps
• Feed execution
results into
story maps
• Sync Story
Story Map
requirements
withTFS
work items

Lessons learned

Collaboration over specification
Documentation after conversation
• Collaborative discovery
• Should trigger new questions
• Infinite number of examples
• Readability and ability to automate

Level of automation
Controller
Business Layer
Data Layer
Model
View
Browser automation
Trigger behavior
through controller
Assert behavior
on model, db, ..
Setup pre-conditions
through service
interfaces
Out-of-process
In-process

Test execution performance
• Grouping tests
– Current WIP
– Completed work
• Database
– In-memory
– Templates for setup
• Parallel execution
• Smart execution order
• Level of automation

Internal vs. external DSL
Example Source: Liz Keogh
https://github.com/lunivore/tictactoe-java/blob/master/
scenarios/com/lunivore/tictactoe/scenarios/Three_in_a_row_wins.java

Non-functional acceptance criteria
Given there are 100,000 users registered on the system
When I create a new account
Then I should be taken to my dashboard within 5ms
When 1000 users are hitting the homepage simultaneously
Then each user should get a response within 2ms
Matt Wynne
http://blog.mattwynne.net/2012/03/13/using-cucumber-for-load-testing

Tools

Gherkin automation tools
www.cukes.info
www.behat.org
www.specflow.org
Ruby, Java, JavaScript, C++
.NET, Mono, Silverlight,WP7
PHP

Books
Bridiging the
Communication Gap
Gojko Adzic
Specification by
Example
Gojko Adzic
Explore It!
Elisabeth
Hendrickson
Exploratory
Software
Testing
James
Whittaker

Summary
Specification-by-example
• Collaborative discovery and specification
• Impact on development and test
• Combine with other agile reqmt methods
• Can be supported by tools

Questions
Got curious about Specification-By-Example?
Learned something new about using SpecFlow?
Christian Hassa,TechTalk
(ch@techtalk.at, @chr99ha)
Join the BoF Session
„Specification-By-Example (ATDD, BDD) in practice“
Today at 15h15 (right after this talk)