Testing im Wandel
Neue Herausforderungen
und Chancen in einer agilen
Welt
Jonas Bandi
Software Architect
TechTalk
jonas.bandi@techtalk.ch
blog.jonasbandi.net
www.techtalk.ch twitter.com/jbandi
Thursday, November 5, 2009

"We're still
figuring this
stuff out.
All of us."
- Jay Fields
Thoughts on developer Testing:
http://blog.jayfields.com/2009/02/thoughts-on-
developer-testing.html
Thursday, November 5, 2009

My point of view
• Software Architect at TechTalk
• TechTalk is a software development
and consulting company with ~60
people located in Vienna, Budapest
and Zürich.
• The main business is building
custom IT solutions for clients in
Austria and Switzerland.
• Projects of TechTalk involve
development efforts of up to
several thousand person days and
team sizes of up to ten persons.
Thursday, November 5, 2009

Once upon a
time...
... in the golden
Age of the
Waterfall.
Thursday, November 5, 2009

... and they lived happily ever after!
Thursday, November 5, 2009

Welcome to the real world!
Thursday, November 5, 2009

... the golden age is over!
Thursday, November 5, 2009

What did really change?
49% 24%
23% 44%
32%
28% sucessful
challenged
failed
2000 2008
Project success
Chaos Report, Standish Group
Thursday, November 5, 2009

What is the problem?
Thursday, November 5, 2009

There are many!
Thursday, November 5, 2009

Focus: Requirements
Thursday, November 5, 2009

What defines success?
Software development is about
delivering business value!
Thursday, November 5, 2009

What is being used?
19% 16% Never
Seldom
13% Occasionally
45% 7% Often
Always
CHAOS Report 2000, Standish Group
Thursday, November 5, 2009

Providing Business Value?
• Only 20% of all delivered features are
actually used?
• Something is wrong!
• We can profit tremendously when we
improve in requirements gathering and in
transporting them successfully to the
implementation
Thursday, November 5, 2009

Where Errors are introduced
Requirements
27% Design
56% Code
7% Other
10%
Source: James Martin, An Information Systems Manifesto
Thursday, November 5, 2009

Fighting Errors?
• Traditional Developer Testing focuses on
code and maybe design
• Testing requirements is hardly ever a topic
• We can profit tremendously when we
improve in requirements gathering.
Thursday, November 5, 2009

Detailgrad
Vision Verständnisgrenze
Actor/Stakeholder
Lastenheft
Pflichtenheft
Designdokument
Kein Feedback von
Stakeholdern möglich! Code
Manual Test Automated
„Usage“ Test
Zeit
Waterfall Project
Thursday, November 5, 2009

Costs to fix defects rise
exponentially over time
The longer a
defect stays in the
development
process, the more
expensive it is to
fix
Thursday, November 5, 2009

Agile to the rescue!
Thursday, November 5, 2009

Detailgrad
Verständnisgrenze
Actor/Stakeholder
Actor/
Business
Vision Stakeholder
Projektstart
goals
goals
Product
Epics Stories
Backlog
Sprint Planning Preparation
Sprint Acceptance
Sprint 1-n
Stories
Sprint Planning Backlog Criteria
Sprint Implementation Acceptance
Kurze Feedbackzyklen Test
Definition
mit Stakeholdern Code
Manual Test/ Automated
Demo Test
Zeit
Iterative Project
Thursday, November 5, 2009

The benefits
• Constant feedback and priorization
by the Stakeholder 19%16%
• Build those features that are 45% 7%
13%
needed and useful
• Features have a much shorter 27%
through-put time
7%
• Allows validation of requirements 56% 10%
Thursday, November 5, 2009

Wait...
... isn’t this
presentation
supposed to be
about testing?
Thursday, November 5, 2009

"I believe that the hardest part of software
projects, the most common source of
project failure, is communication with the
customers and users of that software."
- Martin Fowler, DSL Book
Thursday, November 5, 2009

It is all about communication!
Thursday, November 5, 2009

Verständnisgrenze
Actor/Stakeholder
Actor/
Business
Vision Stakeholder
Projektstart
goals
goals
Product
Epics Stories
Backlog
Sprint Planning Preparation
Sprint Acceptance
Sprint 1-n
Stories
Sprint Planning Backlog Criteria
Sprint Implementation Acceptance
Kurze Feedbackzyklen Test
„Vertrauenswürdige Definition
mit Stakeholdern Code
Kommunikation“
Manual Test/ Automated
Demo Test
Zeit
Enable communication
Thursday, November 5, 2009

How?
Thursday, November 5, 2009

Catch bugs in requirements!
Thursday, November 5, 2009

Thursday, November 5, 2009
Cost
RE
Analysis & Design
Requirements Engineering
Implementation
Testing
Deployment Analysis & Design
Implementation
Time Testing
RE
Analysis & Design
Implementation Deployment
Testing
Deployment

How?
Thursday, November 5, 2009

Now...
... finally testing!
Thursday, November 5, 2009

Requirements == Tests
“As formality increases, tests and
requirements become indistinguishable. At the
limit, tests and requirements are equivalent.”
- Equivalence hypothesis, Robert C. Martin
• Traditionally this is not recognized because RE and
Testing are set up as different disciplines
• But they are about the same: how the system is used
• Wouldn’t it make sense to leverage some synergies?
Thursday, November 5, 2009

Ideally Tests ...
• … describe what the system should do
• … are business readable
• … establish a common vocabulary
• … capture a shared set of examples
• … enable communication
• … are bound to the implementation
• … are executable
... express shared understanding
Thursday, November 5, 2009

Building a shared understanding
- Scripts
- Test Data
Mapping
- Acceptance
Criteria
Business
Business
- Specification - Scripts
Tester Tester
- Use Cases - Test Data
- User Stories - Acceptance
Criteria Shared
? - Specification
Understanding
- Use Cases
Mapping - User Stories
- Code Mapping - Code
- Unit Tests - Unit Tests
- Test Data - Test Data
Developer
Developer
Thursday, November 5, 2009

Finding a common abstraction
Shared
Business Developer Tester Understanding
Business
Tester
Developer
Thursday, November 5, 2009

Shared Understanding
• Terminology and abstractions of business
• Concrete enough for developers
• Business readable
• Bound to implementation
• Consistent
Thursday, November 5, 2009

Specification by example
• Abstract requirements and specifications are not a good tool
for communication. Real-life examples are much better.
• We mostly communicate by examples
• Usually examples are not formalized and not shared
Thursday, November 5, 2009

Examples
• Enable developers and testers to
communicate efficiently with business
people
• Provide enough information for developers
to implement and for testers to verify
• Realistic examples contain precise
information and ask for precise answers
• Are a way to test specifications!
Thursday, November 5, 2009

A new tool!
Thursday, November 5, 2009

Agile Acceptance Testing
Acceptance Test Driven
Development
Behavior Driven
Development
Thursday, November 5, 2009

Demo
Thursday, November 5, 2009

Agile: new challenges!
Thursday, November 5, 2009

Challenges
• Effort
• Regression
• Definition of Done
• Streaming Requirements
• Agile Planing
• Authoritative Source of Information
Thursday, November 5, 2009

Effort
Thursday, November 5, 2009

Effort
• Potentially shippable software each iteration
• Testing everything each iteration
• A lot of iterations -> a lot to test!
• Consequence: Automation is a must!
Thursday, November 5, 2009

Regression
Thursday, November 5, 2009

Regression
• Potentially shippable software each iteration
• High velocity, changes are embraced
• How to guarantee quality?
• Consequence: Automation is a must!
Thursday, November 5, 2009

Definition of Done
Thursday, November 5, 2009

Definition of Done
• When are we finished?
• Testing is an integral part of the development
process and not a separate discipline
• Implementation and testing have to work together
on a fine grained level
• There has to be an efficient communication between
implementation and testing
• Consequence: Relation between implementation and
testing has to be redefined!
Thursday, November 5, 2009

Streaming Requirements
Thursday, November 5, 2009

Streaming Requirements
• Not all of the requirements are delivered upfront
• Requirements are delivered just in time
• Incrementally reveal details just-in-time
• Pull instead of push
• Embrace Change
• Consequence: Requirements providers
(stakeholders) must find a new way to deliver
requirements
Thursday, November 5, 2009

Agile Planning
Thursday, November 5, 2009

Agile Planning
• Goal-driven instead of scope-driven
• How to plan something when the scope is
uncertain?
• Managing changes and uncertainty
• You don’t know what can be tested when...
• Consequence: Testers and developers have to
interact on a fine grained and self organized level
Thursday, November 5, 2009

Authoritative Source of Information
Thursday, November 5, 2009

Authoritative Source of
Information
• Avoid excessive, disconnected artifacts. The source code is
the artifact that ultimately matters.
• No specification, no documentation doesn’t work either!
• Communication is more important than documentation
• But people as the only source of information is not an
option
• Consequence: Find a way for efficient specifications and
documentation that is connected to the source code.
Thursday, November 5, 2009

"We're still
figuring this
stuff out.
All of us."
- Jay Fields
Thoughts on developer Testing:
http://blog.jayfields.com/2009/02/thoughts-on-
developer-testing.html
Thursday, November 5, 2009

Müssen Entwickler Braucht es noch Tester?
jetzt Testen?
JA!
Thursday, November 5, 2009

Entwickler...
• Bindet Beispiele and die
Implementierung
• Worauf gebunden wird ist
flexibel!
Thursday, November 5, 2009

Tester ...
• Bringen Beispiele in eine
testbare Form
• Kollaboration mit Stakeholders
• Mindestens ein Beispiel pro
Feature
• Economy of Scale
• Entwickeln die Tessprache
• Nutzung auch für manuelle
Tests
Thursday, November 5, 2009

Manuelle
Tests ...
• Zur Validierung von Main-Flows
und Integration
• Demo an Stakeholder
• Geringere Anzahl
• Unterstützt durch „Testsprache“
• Automatisierung aufwändiger!
Thursday, November 5, 2009

Aufwand?
• Ca. 15% des Gesamtaufwands für
Testspezifikation
-> 1 Tester versorgt 6-7 Entwickler
• Entwickler: 10-25% Aufwand für
Binding
-> Synergien
Thursday, November 5, 2009

Wartbarkeit?
• Richtiger Abstraktionsgrad
• Wiederverwendbarkeit von Bindings
• Ebene der Bindings
Thursday, November 5, 2009

Why are tests
important?
"Nothing makes a system
more flexible than a
comprehensive suite of
tests! Far above good
architecture and good
design!"
- Robert C. Martin, Oredev, 2008
http://archive.oredev.org/topmenu/video/keynotebobmartin.4.5a2d30d411ee6ffd28880002007.html
Thursday, November 5, 2009

Testing is more than QA!
• Regression Tests enable change
• TDD: Drive the design through tests
• Tests are examples that enable customer interaction
• Tests are examples that can be the specification
• Tests are examples that document the system
• ATDD: Drive the specification through acceptance
tests
• Tests can be the definition of ‘done’
Thursday, November 5, 2009

The Topic is Hot!
www.cukes.info www.concordion.net
www.fitnesse.org www.specflow.org
JBehave / NBehave
http://studios.thoughtworks.com/agile-test-automation
Thursday, November 5, 2009

Resources
The RSpec Agile Testing
Book
Bridging the
communication gap
Thursday, November 5, 2009

BDD is a Mindset not a
Tool!
Thursday, November 5, 2009

Viel Erfolg beim Einsatz von
Behavior Driven
Development und Agilen
Akzeptanztests!
Jonas Bandi
Software Architect
TechTalk
jonas.bandi@techtalk.ch
blog.jonasbandi.net
www.techtalk.ch twitter.com/jbandi
Thursday, November 5, 2009