Executable Specifications Agile Palooza

Executable
Speciﬁca/ons
Documents
for
So7ware
that

Chris
Sterling
Partner
at
Sterling
Barton
Cer/ﬁed
Scrum
Trainer
Agile
Consultant

Thursday, January 14, 2010 1

Executable
Speciﬁca/ons
-‐
Topics
to
Discuss
• What
are
Executable
Speciﬁca/ons?
• Why
Create
Them?
• Acceptance
Tests
• When
to
Use
• An
Approach
with
Example
• Available
Tools

2
Copyright © 2010 Sterling Barton. All rights reserved.


The
second-‐most
detailed
speciﬁca/on
of
the
customer’s
request

WHAT
ARE
EXECUTABLE
SPECIFICATIONS?

3


Executable
Specifica/ons

• Unambiguous
defini/on
of
desired
so7ware
behavior
• Executable
documenta/on
• Repeatable
and
specific
tests
• Regression
tests
to
validate
exis/ng
so7ware
behavior
• The
second-‐most
detailed
specifica/on
of
the

customer’s
request

4


Building
the
“right”
so7ware

WHY
CREATE
EXECUTABLE
SPECIFICATIONS?

5


Increasing
Cost
of
Release
Stabiliza/on

Periods

500,000

375,000

250,000

125,000

Release 1
Release 2
Release 3 0
Release 4
Release 5
Release 6

Cost of Fixing Defects Cost for Feature Dev

6


Costly
Regression
Test
Phases

7


Managing
So7ware
Debt
–
an
Overview

8


Eﬀect
of
Project
Constraints
on
Quality

9


How
does
the
Team
know
they
have
implemented
desired
behavior
successfully?

ACCEPTANCE
TESTS

10


Agile
tes/ng
happens
at
two
levels

• Acceptance
Tests
tell
us
whether
the

system
does
what
the
customer

expects
(“building
the
right
code”)

• Programmer
Tests
deﬁne
whether
the

system
does
what
the
developers

expect
(“building
the
code
right”)

11


Acceptance
Tests

• Are
also
called
customer
tests
or
func/onal
tests
• Tell
us
whether
the
system
does
what
the
customer

expects
• Enable
Developers
to
know
they’ve
sa/sﬁed

requirements
• Can
be
run
automa/cally
at
any
/me
by
anyone
• Helps
us
build
the
“right”
so7ware
• “Running,
Tested
Features”*

* “A Metric Leading to Agility” – Ron Jeffries
http://www.xprogramming.com/xpmag/jatRtsMetric.htm

12


User
Stories
Story 14
As a customer I want to check my
order status online so that I can know
when to expect my package

13


User
Stories
Story 14
As a customer I want to check my
order status online so that I can know
when to expect my package

A small piece
of business
value that can
be delivered in
an iteration
13


What
is
a
User
Story?*
• CARD
– Token
represen4ng
the
requirement.
It's
used
in
planning.

Notes
are
wriDen
on
it,
reﬂec4ng
priority
and
cost
• CONVERSATION
– The
requirement
itself
is
communicated
from
customer
to

programmers
through
conversa4on
(The
conversa4on
is

largely
verbal,
but
is
oMen
supplemented
with
documents)
• CONFIRMATION
– The
conﬁrma4on
provided
by
the
acceptance
test
is
what

makes
possible
the
simple
approach
of
card
and
conversa4on
– When
the
conversa4on
about
a
card
gets
down
to
the
details

* “Essential XP: Card, Conversation, Confirmation” – Ron Jeffries
http://www.xprogramming.com/xpmag/EXPCardConversationConfirmation.htm
14


Conﬁrma/on
through
Acceptance
Tests

• Product
Owner
makes
ﬁrst
pass
at
Acceptance
Criteria

• During
Itera/on
Planning
Acceptance
Criteria
are
discussed
• Final
Acceptance
Criteria
for
each
User
Story
is
product
of

nego/a/on
between
Delivery
Team
and
Product
Manager
• Should
be
short,
easy
to
understand
statements

Story 14
As a customer, I want to check
my order status online so that I
can know when to expect my
package

15


Conﬁrma/on
through
Acceptance
Tests

• Product
Owner
makes
ﬁrst
pass
at
Acceptance
Criteria

• During
Itera/on
Planning
Acceptance
Criteria
are
discussed
• Final
Acceptance
Criteria
for
each
User
Story
is
product
of

nego/a/on
between
Delivery
Team
and
Product
Manager
• Should
be
short,
easy
to
understand
statements

Story 14 Acceptance Criteria
•View status as “waiting for pickup”,
As a customer, I want to check “en route” or “delivered”
my order status online so that I •Date of each step in route
can know when to expect my •Estimated time of delivery
package

15


Take
on
the
Role
of
a
User
-‐
“What

should
the
so7ware
do
next
for
me?”

This
ques/on
helps
you
to
decide
what
the

next
acceptance
test
should
model

16


Acceptance
Test-‐Driven
Development

17


Legacy
so7ware,
so7ware
test
automa/on,
and
commercial
oﬀ-‐the-‐shelf

WHEN
TO
USE
EXECUTABLE
SPECIFICATIONS?

18


Tradi/onal
balance
of
tests

Manual GUI Easy to Create
Acceptance Very familiar – what we always do
Tests Typically tedious
How do we know coverage?

Need Automation specialists
Automated Automation good for performance
GUI Tests Seems like we always rewrite
Sometimes fragile
Unit
Tests What is Development testing?
How do we know what these are?
How do we know when they fail?

(Originally discussed by Mike Cohn)

19


Mike
Cohn’s
Tes/ng
Pyramid

Small number
Automate most
GUI
Acceptance
Tests
Drive development and acceptance

Executable
Specifications
Do the most
Create Test Driven Design

Unit Tests

(Also “borrowed” from Mike Cohn)

20


Working
with
Legacy
So7ware

Define
Acceptance
Criteria for
Requirement

21


Working
with
Legacy
So7ware

Define Analyze What
Acceptance Might Be
Criteria for Affected by
Requirement Requirement

21


Working
with
Legacy
So7ware

Define Analyze What Write
Acceptance Might Be Executable
Criteria for Affected by Specifications
Requirement Requirement for How it
Works Now

21


Working
with
Legacy
So7ware

Works Now

Write Failing
Executable
Specifications
for How it
Should Work

21


Working
with
Legacy
So7ware

Works Now

Carefully Write Failing
Modify Source Executable
Code to Specifications
Implement for How it
Requirement Should Work

21


Working
with
Legacy
So7ware

Works Now

Execute Carefully Write Failing
Executable Modify Source Executable
Specifications Code to Specifications
to Verify Implement for How it
Change Requirement Should Work

21


Working
with
Legacy
So7ware

• Focus
on
Wri/ng
Executable
Specifica/ons
for
New

Requirements
and
Func/onality
that
may
be
Affected
• Stabilize
Legacy
So7ware
with
Black-‐box
Tests

(Executable
Specifica/ons)
• Incrementally
Improve
the
Internal
So7ware
Design
• Try
to
Iden/fy
where
Programmer
Tests
can
be
Added
• Execute
Regression
Test
Suite
that
Includes
Executable

Specifica/ons
Frequently
(more
than
once
per
day
if

possible)

22


So7ware
Test
Automa/on

Manual Test Script
Login as
Step 1
Administrato
r
Create
Step 2
Order with 1
Item
Click “Check
Step 3
Out” Button
Verify Item in
Step 4
Shopping
Cart

23


So7ware
Test
Automa/on

Manual Test Script
Login as
Step 1
Administrato
Automate r
Create
Step 2
Order with 1
Item
Click “Check
Step 3
Out” Button
Verify Item in
Step 4
Shopping
Cart

23


Commercial
Oﬀ-‐the-‐Shelf
(COTS)

Version 1:
COTS
Software

24


Commercial
(COTS)

Create Version 1:
Executable
Specifications for COTS
Configurations &
Customizations
Software

24


Commercial
(COTS)

Create Version 1:
Executable
Configurations &
Customizations
Software

Version 2:
COTS
Software

24


Commercial
(COTS)

Create Version 1:
Executable
Configurations &
Customizations
Software

Execute Existing Version 2:
Executable
Specifications to COTS
Find Upgrade
Path
Software

24


Commercial
(COTS)

Create Version 2:
Executable
Configurations &
Customizations
Software

24


Walk
through
the
crea/on
of
executable
speciﬁca/ons
and
how
to
execute
them
eﬀec/vely

AN
EXAMPLE
APPROACH

25


A
simple
example
of
a
FIT
test

*Source: http://fit.c2.com/

26


Fit
Workﬂow*

• Starts
with
whiteboard
conversa/on
• Customer
(SME,
business
person,
product
manager,

etc.)
informally
describe
new
feature
• Programmers
and
testers
ask
ques/ons


27


Fit
Workﬂow*

• Customer,
working
with
delivery
team,
reﬁnes

examples
into
tables
• Use
business-‐friendly
tools,
such
as
Microso7
Excel
and

Word,
to
capture
test
tables


28


Fit
Workﬂow*

• Delivery
team
suggest
addi/onal
areas
to
cover


29


Fit
Workﬂow*

• Delivery
team
format
tables
for
use
with
Fit


30


Fit
Overview


31


Fit
Workﬂow*

• Delivery
team
creates
Fit
“ﬁxtures”
(small
piece
of
code

that
translates
test
tables
into
execu/on
tests
against

the
so7ware)


32


Fit
Workﬂow*

• Execute
Fit
document
• Click
to
edit
Master
text
styles
against
so7ware
• At
ﬁrst
some
tests
should

be
failing
(red)


33


Fit
Workﬂow*

• Delivery
team
• Click
to
edit
Master
text
styles
collaborates
with

customer
to
incrementally

enhance
test
tables
• Delivery
team
implements

so7ware
to
meet
test

execu/on
(green)


34


Fit
Workﬂow*

• Document
is
kept
for
regression
tes/ng
• Document
is
included
in
automated
build
to
ensure

everything
keeps
working

• As
ques/ons
arise
about
func/onality
an
example
is

added
and
Fit
reports
the
answer


35


Con/nuous
Integra/on

36


Organizing
FIT
tests

• Maintain
suite
of
regression
tests
from
past
itera/ons

that
always
pass
• Run
“regression”
tests
with
build.
• Maintain
a
suite
of
“in
progress”
tests
for
the
current

itera/on
– Begin
the
itera4on
with
all
tests
failing
– End
the
itera4on
with
most
tests
passing
• At
the
end
of
the
itera/on,
move
newly
passing
tests

into
regression
suite
– Beware
the
Fitnesse
“Refactor/Move”
command
37


Automated
Regression
Tes/ng

Application
Under
Development

38


Automated
Regression
Tes/ng

Application
Under
Development

New
Feature

38


Automated
Regression
Tes/ng

Application
Under
Development

New Design Changes
Feature

38


Automated
Regression
Tes/ng

Application
Under
Development

Automated
Design Changes Regression
New Test Run
Feature

38


Cost
reduc/on
using
Fit
for
test
automa/on
and
data
conversion

A
FIT
CASE
STUDY

39


Manual
Regression
Tes/ng

• Tes/ng
was
taking
75
person
hours
during
2
full
test

runs
consis/ng
of:
– Comprehensive
manual
regression
tes4ng
– Data
conversion
and
valida4on
• Cost
for
tes/ng
was
$17,000
each
itera/on

40


Introducing
Fit
into
Tes/ng
Process

• A7er
8
itera/ons
team
had
introduced
healthy
amount

of
Fit
ﬁxtures
and
automated
tests
• Reduced
70+
hour
test
run/me
down
to
6
hours
which

now
included:
– Fit
automated
regression
tes4ng

– Data
conversion
and
valida4on
automated
with
Fit
ﬁxtures

• Reduced
cost
of
tes/ng
each
itera/on
from
$17,000
to

$7,000

41


Tools
that
Teams
can
use
and
what
context
they
are
best
used
in

AVAILABLE
TOOLS

42


Table-‐based
Acceptance
Tes/ng
Tools

• Fit
–
hlp://ﬁt.c2.com/

• FitNesse
–
hlp://www.ﬁtnesse.org/
• RobotFramework
-‐
hlp://code.google.com/p/
robonramework/

43


Programmer
Acceptance
Tes/ng
Tools

• Cucumber
-‐
hlp://cukes.info/

• Rspec
-‐
hlp://rspec.info/

• JBehave
-‐
hlp://jbehave.org/

• JMeter
-‐
hlp://jakarta.apache.org/jmeter/

• xUnit
(JUnit/NUnit/CppUnit/etc…)

44


Web
UI
Agile
Tes/ng
Tools
• Open
Source
Web
UI
Tes/ng
tools
– StoryTestIQ
– WATiR
– Selenium
– Canoo
Web
Test
• GUI
tests
tend
to
be
more
brille
• Tests
wrilen
and
maintained
incrementally
• Most
of
these
tools
have
code
integra/on

45


What
about
tradi/onal
tes/ng
tools?

• Why
don’t
we
use
commercial
products
such
as
SilkTest,

TestDirector,
QuickTest
Pro,
etc.
for
Agile
tes/ng?
– Expensive
– Automated
tools
are
record-‐and-‐playback;
briDle
– Ties
our
tests
to
the
UI
implementa4on
– Manual
tools
(TestDirector)
take
too
long
to
run.
– Work
ﬁne
as
an
interim
strategy
(especially
if
you
already

have
the
licenses)
– Consider
adding
open
source
Agile
Tes4ng
tool
as
a

component
of
your
tes4ng
strategy

46


Ques/ons
and
Answers

THANK
YOU

47


Presenter:
Chris
Sterling
Partner
at
Sterling
Barton
• Technology
Consultant,
Cer/ﬁed
Scrum

Trainer
and
Agile
Coach
• Consults
on
so7ware
technology
across
a

spectrum
of
industries
• Consults
organiza/ons
on
Agile

development,
management,
and

enterprise
prac/ces
• Founder
of
Interna/onal
Associa/on
of

So7ware
Architects
(IASA)
Puget
Sound

chapter
• University
of
Washington
Lecturer:
Agile

Developer
Cer/ﬁcate
Program

Email:
chris@SterlingBarton.com
Our
Blog:
hlp://www.getngagile.com
Follow
me
on
Twiler:
@csterwa

48


Executable Specifications Agile Palooza

Recommended

Recommended

More Related Content

More from Chris Sterling

More from Chris Sterling (20)

Recently uploaded

Recently uploaded (20)

Executable Specifications Agile Palooza