Michael mahlberg exploratory-testing-the_missing_half_of_bdd

Michael Mahlberg, 05.06.2012

Exploratory Testing –
The missing half of BDD?

Tuesday, June 5, 2012

Testing?


AGAIN?


It’s not about Rails

http://www.ﬂickr.com/photos/chris_gin/2490980153/


Several Stories

BDD Speciﬁcation Design by
TDD by Example Contract

Defensive
Programming

Software-
Development

Exploratory
Testing


Paul Carvalho

@can_test


Michael Bolton

No, Not this one


Michael Bolton

@michaelbolton


Markus Gärtner

@mgaertne


Players in the Game

• James Bach
• Michael Bolten (not the singer)
• Cem Kaner
• Paul Carvalho


Ressources
Human Resource
Paul Carvalho http://swtester.blogspot.de/

Michael Bolton http://www.developsense.com/blog/

James Bach http://www.satisﬁce.com/blog/

Elisabeth Hendrickson http://testobsessed.com/

Markus Gärtner http://www.shino.de/blog/


Ressources by James Bach
Topic Ressource
SBTM Session Based http://satisfice.com/articles/sbtm.pdf
Test Management
http://people.eecs.ku.edu/~saiedian/
Exploratory Testing Teaching/Fa07/814/Resources/
exploratory-testing.pdf

Hands-on Exploratory http://www.satisfice.com/tools/
procedure.pdf
Testing
Rapid Software Testing http://www.satisfice.com/rst.pdf


The basic misunderstanding

• <X> is necessary but not sufﬁcient


An example from agile
software development


An example from agile
software development

Working Software
over

Comprehensive Documentation


Derived Assumption:
Working software is sufﬁcient


Reality:
Working software is necessary,
but not sufﬁcient!


So what’s the problem?

■ Software testing, the process of verifying that a
software program works as expected
Source: http://en.wikipedia.org/wiki/Test



... but ...



■ Test
[...]
Etymology
From Old French test (“an earthen vessel,
especially a pot in which metals were tried”)
Source: http://en.wiktionary.org/wiki/test


What do test pilots do?


Pushing the envelope


Checking things


Scripted testing

With a script, you miss the
same things every time.
Exploratory Testing @ QUEST 2008" Copyright © 2008 Cem Kaner


Defensive programming
- Check all input parameters
- Ascertain availability of resources
etc.

Defensive programming defends against the currently
impossible (c2 Wiki)


Design by Contract
Betrand Meyer
Since 1986
Programming language: Eiffel


Design by Contract
Betrand Meyer
Since 1986
Programming language: Eiffel

Preconditions
Postconditions
Invariants


TDD

• Unit Tests
• Black-Box Tests
• White-Box Tests
• RSpec etc.


BDD

• Acceptance Tests
• (Mostly) Black-Box Tests
• Based on Scenarios


ATDD

• Bigger gap between test creation and
execution
• Exclusively Black Box Testing


An adage from agile software
development


An adage from agile software
development

INVEST in stories and
come up with SMART tasks


INVEST
Independent
Negotiable
Valuable
Estimatable
Small
Testable


SMART
Speciﬁc
Mesuarable
Achievable
Realistic
Tangible


Independent
Speciﬁc
Negotiable
Mesuarable
Valuable
Achievable
Estimatable
Realistic
Small
Tangible
Testable


Acceptance Criteria
• To accomodate pre-conditions
Acceptance Criteria can be expressed as
• Given [Precondition]
When [Actor and Action]
Then [Observable Result]
• Naresh Jain Slide 29

Speciﬁcation by Example

... It is the result of a research on
how teams all over the world specify,
develop, test and deliver the right
software, without defects, in very
short iterative delivery cycles. ...


What’s left for the
Testers?


DbC vs. TDD
Design by Contract
implies automated defensive
programming
pre- and postconditions and Test Driven Design is
invariants are checked at mostly applied during design-
runtime time.
Expensive checks are Tests are executed whenever
inhibitive appropriate - but usually not
in the production
environment
Expensive checks are
possible but unwarranted


Checking vs. Testing
Checking is something that
we do with the motivation of
confirming existing beliefs

Testing is something that we
do with the motivation of
finding new information.

http://www.developsense.com/blog/2009/08/testing-vs-checking/


Test Driven Development
is an oxymoron

Check Driven Development?


The Test-Cycles


Red-Green-Refactor
Failing
Test

Simplify Coding
Remove Redundancy


Exploratory Testing

From: Paul Carvalho
http://swtester.blogspot.de/2012/05/what-is-exploratory-testing.html


So what is Exploratory Testing?

"Exploratory Testing is simultaneously
learning about the system while designing
and executing tests, using feedback from
the last test to inform the next."

Elisabeth Hendrickson, @testobsessed
Quoted by Paul Carvalho


Cone of uncertainty

Agile and Iterative Development: A Manager's Guide, Craig Larman 2003


Cone of uncertainty


Cone of test-cases

http://swtester.blogspot.de/2012/05/what-is-exploratory-testing.html


Agile testing quadrants

http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/


The Prime Directive
Be Thoughtful and Methodical
Throughout the test procedure, as you complete the tasks, you
have lots of freedom about how you do the work. But you must
work methodically, and follow the procedure. In the course of
creating the result for each task, you’ll ﬁnd that you have to
make a lot of guesses, and some of them will be wrong. But you
must think. If you ﬁnd yourself making wild and uneducated
guesses about how the product works, areas of instability, or
anything else, stop and talk to the Test Manager.

James Bach, 08/26/99, http://www.satisfice.com/tools/procedure.pdf


Scientiﬁc Method
Formulate a question
Hypothesis
Prediction
Test
Analysis


Exploratory Testing
Complete these five tasks:
✓ Identify the purpose of the product.
✓ Identify functions.
✓ Identify areas of potential instability.
✓ Test each function and record problems.
✓ Design and record a consistency verification
test.
Source: http://www.satisfice.com/tools/procedure.pdf
pg. 11 to 22


Identify the purpose of the
product

• With a tangible result (i.e. Document)
• Value for others - veriﬁable etc.
• Value for the tester - builds
understanding


Identify functions

• By actually using (skimming) the product
• Explorative (not based on documentation)
• Tangible output (Function outline)


From James Bach:

http://www.satisﬁce.com/tools/procedure.pdf


Identify areas of potential
instability

• Results:
• Documented areas (before testing)
• Documented data for testing
• Documented Strategies


Test each function and record
problems
• Actually this means: Try to do things the
development teams did not think of.

• Remember: A Problem is just the
difference between the expected behavior
and the observed behavior - whether it is
cause for action has to be decided by the
customer


Design and record a
consistency verification test

• Try to extrapolate if the Software would
run in (slightly) modified environments.

(The original 5 steps are from 1999 (gasp)
and relate to specific windows versions -
the basic requirements still hold through)


Tool Support
Area Tool URL
http://
Recording spector www.spectorsoft.com/

http://
Recording BBTestAssistant www.bbsoftware.co.uk/
products.aspx
http://
Recording Wink www.debugmode.com/
wink/

http://www.sirius-
All in One TestExplorer sqa.com/

http://www.satisﬁce.com/blog/archives/77

Session Based Test Management
• Original description of ET : 1999
• Today : 2012
• SBTM: A way to organize exploratory
testing
• Working in 45 to 90 min Session based
on Charters
• See: http://satisﬁce.com/articles/sbtm.pdf

What are Charters
Counterpart to the use case / user story
Some of the same attributes
- Roles, activities, actors, systems
Examples:
- “Analyze <Product>’s View menu functionality and
report on areas of potential risk.”


Hierarchy


Hierarchy

Ad-Hoc Testing


Hierarchy

Exploratory Testing

Ad-Hoc Testing


Hierarchy

Session Based Testing

Exploratory Testing

Ad-Hoc Testing


Hierarchy
Session Based Test Management

Session Based Testing

Exploratory Testing

Ad-Hoc Testing


ET-Spectrum


ET-Spectrum

Exploratory Testing according to:
http://www.satisﬁce.com/rst.pdf


Can we prove it?
Testing shows the presence,
not the absence of bugs
J.N. Buxton and B. Randell, eds, Software Engineering Techniques,
April 1970, p. 16. Report on a conference sponsored by the NATO
Science Committee, Rome, Italy, 27–31 October 1969.

via http://en.wikiquote.org/wiki/Edsger_W._Dijkstra

Program testing can be used to
show the presence of bugs, but
never to show their absence!
Source: Notes On Structured Programming, 1972, at the end of section 3, On The
Edsger Wybe Dijkstra Reliability of Mechanisms. EWD249 (1970).


TDD
• Technique to drive the design of your code

y, t!
r n
sa cie
s ﬁ
ce uf
e s
N ot
t n
Bu


BDD
• Technique to capture requirements and
drive desired functionality

y, t!
r n
sa cie
s ﬁ
ce uf
e s
N ot
t n
Bu


ET
• Technique to drive the discovery of
unwanted, unexpected and unpredicted
behaviour

y, t!
r n
sa cie
s ﬁ
ce uf
e s
N ot
t n
Bu


About me
“Consulting on software development
processes and software architecture since
the last millennium”
- @MMahlberg

Google me: “Michael Mahlberg”


Discussion?


Flight-Envelope
Photo Credits
http://en.wikipedia.org/wiki/File:PerformanceEnvelope.gif

Testpilots:
http://www.flickr.com/photos/x-ray_delta_one/4106250812/in/photostream/

Spreading junction
http://www.flickr.com/photos/elsie/4164899/in/photostream/

Tokio Tower
http://www.flickr.com/photos/wilhelmja/330965980/in/photostream/

Eiffel Tower
http://www.flickr.com/photos/gcattiaux/2565101112/in/photostream/

Spectrum
http://www.flickr.com/photos/eiriknewth/132859486/in/photostream/

Armor
http://www.flickr.com/photos/nicmcphee/1563082378/in/photostream/

Aerobatics
http://www.flickr.com/photos/ssandars/6827302/in/photostream/

Airbus
http://www.flickr.com/photos/mwanasimba/4097232435/


Michael mahlberg exploratory-testing-the_missing_half_of_bdd

More Related Content

Viewers also liked

Similar to Michael mahlberg exploratory-testing-the_missing_half_of_bdd

More from Michael Mahlberg

Recently uploaded

Michael mahlberg exploratory-testing-the_missing_half_of_bdd