This document discusses specification by example (SBE) as a way to write requirements and acceptance tests together in agile software development. It notes that SBE involves writing examples that serve as both requirements and tests, allowing teams to ensure they have a shared understanding of what needs to be built. The document outlines benefits of SBE such as improved documentation, reduced defects, and easier test automation. It also discusses how SBE can be combined with test-driven development and continuous integration to facilitate continuous acceptance testing.

1. © CGI Group Inc. CONFIDENTIAL
Specification by Example
Requirements in the Agile Process
Chris C. Schotanus - CGI
12 November 2015

2. Let’s Start With Some Quotes…
“The hardest single part of building a software system is
deciding precisely what to build.
- No other part of the conceptual work is as difficult as
establishing the detailed technical requirements
- No other part of the work so cripples the resulting
system if done wrong.
- No other part is as difficult to rectify later” Fred Brooks
“The inability to produce complete, correct, and
unambiguous software requirements is still considered
the major cause of software failure today” M. Dorfman
2 12 november 2015

3. …And What Testers Say
“More than the act of testing, the act of designing
tests is one of the best defect preventers known …
The thought process that must take place to create
useful tests can discover and eliminate problems
at every stage of development” Boris Beizer
“One of the most effective ways of specifying
something is to describe (in detail) how you would
accept (test) it if someone gave it to you.” Bill Hetzel
3
12 november 2015

4. The Agile Process
12 november 2015
7-30 days
24 hours
Sprint Backlog
Potentially Shippable
Product Increment
Source: Adapted from Agile Software
Development with Scrum by Ken
Schwaber and Mike Beedle.
Product Owner
Scrum master
Team member
Stakeholder
4
Backlog Refinement

5. Sprint: avg. 3 user stories + script
STus1
US 1 US 2
RTus1
IT
STus3
US 3
RT
us1-2
IT
ST us
2
DEMO
IT
SIT ORT
Sprint
4 weeks
SprintPlanning
(1–2days)
E2ETest
1week
UserAcc
1week
SprintReview
(1–2days) Buss.
UAT
Production
US = User Story
ST = System test
IT = Itegration test of
user story rsesults
TC = Test Case
RT = Automated Regression
test within sprint
SIT = System Integration test
ORT = Overall RT
UAT = User Acceptance Test
Automation Script creation
Elaborate
US
Program
changes
Elaborate TC´s
System
test
Elaborate
US
Program
changes
Elaborate TC´s
System
test
Aut
Regr
test
Elaborate
US:
Define Test
conditions
Program
changes
Elaborate TC´s
System
test
Activities in an Iteration
12 november 20155

6. Test Driven Development
Automated Testing at Unit Test Level

7. eXtreme Programming’s “Test First” principle
• Create test cases before writing code
• Business representatives write acceptance tests to demonstrate that user
stories are correctly implemented
• Programmers continually write component tests which must run flawlessly
for development to continue
• “We only write new code when we have a test that doesn’t work”
7
12 november 2015

8. Test Driven Development
Although the technique is even older, Kent Beck introduced the name
TDD in 2003. The principle is that the developer starts writing tests which
are used to test the code that the developer will write.
8
Test Driven Development (TDD) is a software development technique!
RED
GREENREFACTOR
TDD
12 november 2015

9. The Biggest Benefits of Test-driven Development
Defect Reduction Functionality that doesn't function provides no business
value. TDD allows us to start with high quality, thus providing real
value.
Faster Feature Time to
Market
TDD allows developers to add innovative features as rapidly as
can be, without damaging the original investment
Improved Focus TDD helps developers think clearly about the task at hand, without
having to continuously hold the whole complex system in their
heads. They avoid over-extrapolation and over-design.
Parallel Efforts Without
Conflict
Since, with disciplined TDD, each developer (or pair of
developers) is expected to run all of the tests in the regression
suite and confirm that 100% pass before committing new changes,
there is much less opportunity to disrupt or destroy someone
else's hard work.
Test Assets During a TDD development process programmers create test
code that is automatically executed and therefor valuable as
regression test ware
Source: Agileinstitute.com
9 12 november 2015

10. However, There are Limitations to TDD
10
TDD leads to high quality software, not necessarily to
the correct software
12 november 2015

11. Specification by Example
A way to kill two birds with one stone

12. The Basics
Examples Tests
Requirements
Can become
Source: Gojko Adzic: Specification by Example
12 november 201512

13. 12 november 201513
High level demand
Business rule/requirement
Test case
Test case
Test case

14. Implementing Acceptance Criteria
Validate that a story has been developed with the functionality the
Product Owner had in mind.
• Test cases fill in the details of the story.
• Write tests before programming – Test first, build next.
• Execute the acceptance test every day
• Test cases are delivered as system documentation.
14
Title: Book a flight
ID: US12a
Scenario: As a member of the Flight Loyalty Club
I can use Flight Miles to pay for a flight
so that I can travel for free
Estimation: 13 points Priority: High
12 november 2015

15. 15
Examples in a DSL (Gherkin)
Example:
Given user "123.456.789.0" with password "abcd" is logged in
And departure city is "AMS"
And destination city is "CDG"
And return trip = “Y”
When departure date is 100 days after today
And return date is 99 days after today
And query is submitted
Then the alert "Date of return flight is incorrect!" is shown
Acceptance Criterion:
The return date must be later than or equal to
the arrival date
12 november 2015
Initial situation
Test action
Expected result

16. Automating the Test Execution
Given user "123.456.789.0" with password "abcd" is logged in
And departure city is "AMS"
And destination city is "CDG"
And return trip = “Y”
When departure date is 100 days after today
And return date is 99 days after today
And query is submitted
Then the alert "Date of return flight is incorrect!" is shown
16
Keyword driven testing:
• Automation scripts per keyword
• Parameters contain test data
12 november 2015

17. Reuse Using tables
Data driven testing
Given user “@User" with password “@Password" is logged in
And departure city is “@Deptcity"
And destination city is “@Destcity"
And return trip “@return”
When departure date is “@Deptdate” days after today
And return date is “@Retdate” days after today
And query is submitted
Then the alert “@Alert” is shown
17
user password Dept
city
Dest
city
return Dept date
(now+#days)
Ret Date
(now+#days)
Alert
123.456.789.0 abcd AMS CDG N 13 - Bookingdate must be 14 days in the
future
123.456.789.0 abcd AMS CDG Y 100 99 Date of return flight is incorrect!"
123.456.789.0 abcd AMS CDG Y 100 101 Dates accepted"
12 november 2015

18. A Somewhat Different layout
Test Cases in standard keyword driven format (CGI TestFrame)
18
Test ConditionUS12a-C2 The return date must be later than or equal to the arrival date
Test case US12a-C2-T2 Arrival date is later than return date
Login to F&H
Fresh & Honest Club number Password
login f&h club 123.456.789.0 abcd
Search a flight with invalid dates
Departure City Destination City Departing Returntrip Returning
search flight ams cdg &Date(),100 Yes &Date(),99
Message
check message Date of return flight is incorrect!

19. Create
software
Refactor
TDD
Write a
failing
test
Make the
test pass
SbE
Write an
acceptance
test
Run the
acceptance
test
Specification by Example
19 12 november 2015
TDD combined with acceptance testing

20. Benefits of Specification by Example
20

21. Overall Benefits
• Stakeholders and team jointly decide on how to solve the requirement
Common understanding
• The software is accepted if all tests are passed
Clear acceptance criteria
• The test cases are the documentation. In case of a change, the test cases
will be changed first
Living Documentation
• There are off the shelf tools (open source as well as COTS), that are able
of automating gherkins
Easy automatable test cases
• The test cases are executed each and every cycle
Regression test implicitly maintained
12 november 201521

22. The Greatest Benefits of SbE in More Detail
It is not about “testing” SbE is a design approach, another tool in the box to facilitate goal oriented
communication and get everybody on the same page.
No more outdated documents Another benefit is that there are less documents in the true sense of the word.
Test cases and specs are the same and always in sync
Easy automatable test
execution
SbE Specs (Examples) are written in a format that is directly automatable, right
from the first execution
Automated regression test
instantly available
Every sprint will deliver test cases and test scripts. It is just a matter of selection
of Regression Test cases (depending on the product risks that were defined)
No more throwing artifacts
over the wall
Since everyone is involved during refinement of the user stories, there will be a
common understanding of what to achieve
from a project focus: First-Time-Right
22 12 november 2015
from a business focus: we get what we need
The system just works There are almost no findings (either because of errors or changes) after the
software is released, so First-time-Right
Focus is on the problem not
the solution
SbE specs must be written in a way that really describes the business problem to
be solved and not how the user interface of the application will be used.
Faster time to market The application of SbE leads to an overall reduction of project duration due to less
misunderstanding

23. SbE vs “Conventional” Agile Development.
12
Definition
User Stories
Refine-
ment Development Test
Demo
Test case design
Definition
User Stories
Test case
design =
refinement
Less effort Faster TTM
Demo
Product Backlog Sprint Acceptance
Continuous
Development and Test
12 november 2015

24. CI and Specification by Example
Continuous Acceptance Testing
24

25. Practices of Continuous Integration
• Maintain a Single Source Repository.
• Automate the Build
• Make Your Build Self-Testing
• Everyone Commits To the Mainline Every Day
• Every Commit Should Build the Mainline on an Integration Machine
• Fix Broken Builds Immediately
• Keep the Build Fast
• Test in a Clone of the Production Environment
• Make it Easy for Anyone to Get the Latest Executable
• Everyone can see what's happening
• Automate Deployment
12 november 201525

26. So, CI means that:
• The programmer will unit test the branch
• The branch will be integrated into the trunk
• On the build server the integrated software
is built and tested
And this many times a day!
(Btw, Amazon deploys new software to production
every 11.6 seconds.
The source for this little nugget is a talk from Jon Jenkins of Amazon,
speaking at Velocity 2011)
12 november 201525
Automate the Build
Make Your Build Self-Testing

27. The CI Process (Simplified)
12 november 201526
TDD SBE

28. Conclusion
27

29. The Advantages of Specification by Example
Specification by Example has numerous advantages:
• Improved, living documentation, easy requirements management
• Common understanding by anyone involved (business & IT)
• Less production failures, first time right delivery
• Overall shorter development and QA cycles
• No “that’s not what I meant” complaints anymore, so no rework and
first-time-right delivery
• Easy automatable and therefor unmissable in Continuous Integration
28 12 november 2015

30. And Remember
Transformation into a solution of even the SMART-est requirements
may fail due to misunderstanding and assumptions. This often
leads to unhappy users or even high problem solving cost.
When business representative, product owner, developer and tester talk
to each other and create acceptance tests, misunderstanding and
assumptions tend to be discovered.
But beware: if you apply Specification by Example just as a test
approach, you’ll loose these advantages
29 12 november 2015
chris.schotanus@cgi.com