Software quality - no more bugs!

1Copyright © 2012 Retalix |
|
NO MORE BUGS!
By: Arnon Axelrod
SOFTWARE QUALITY

About me

About you
• Name
• Company
• Role
• Number of people in the project
• Average release cycle
• % of man-months dedicated to bugs finding and fixing
• From 1 to 10 – how are you satisfied with the quality of
the software you deliver?

Definition of Done

What is quality?

The quality pyramid
• Intuitive, sleek, “sexy”,
enjoyable…
• User friendly, HelpUseful
• Stable
• Supportable
• Secure
Reliable
• Useful features
• Solves a user’s
problem
Usable

The quality pyramid – “reality”
enjoyable…
• User friendly, Help
• Stable
• Supportable
• Secure
Reliable
• Useful features
problem
Usable

The quality pyramid
enjoyable…
• User friendly, HelpUseful
• Stable
• Supportable
• Secure
Reliable
• Useful features
problem
Usable
SbE / BDD / ATDD

Bugs

What is a bug?

What is a bug?
Wikipedia:
“A software bug is an error, flaw, mistake, failure, or fault in
a computer program or system that produces an incorrect
or unexpected result, or causes it to behave in
unintended ways”

Why there are bugs?
• Expectations gaps • Miscommunication
• Coding errors
• Miscommunication
between teams or team
members
• Misunderstanding the
business needs
• Incomplete or incorrect
definition of the expected
results
• Complexity
• Misuse of tools
• Bugs in tools

Bugs - conclusion
By improving the communication and expectation
coordination – we can prevent many bugs

Problems with our current practices

Waterfall

Is Agile the answer?

Automated Testing

Unit Testing

Introduction to
Specification by
Example

The Agile Manifesto
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.

Different names
• Specification by Example
• ATDD (Acceptance Test Driven Development)
• Agile Acceptance Testing
• Example Driven Development
• Executable Specifications
• BDD (Behavior Driven Development)

How does it work?
Specifications
Tests
Examples of
user
scenarios

User Stories
“are a promise for a future conversation”
– Ron Jeffries
As a <role/persona>
I want <goal/desire>
so that <benefit>.
In order to <receive benefit>
as a <role/persona>
I want <goal/desire>
As a user closing the app
I want to be prompted to save my
work
so that I won’t loose all I’ve done
so far
In order not to loose all I’ve done
so far
as a User closing the app
I want to be prompted to save my
work

Deriving the Solution from the Goal

Specifying Collaboratively

• Attendees: BA, Developer, Tester
• Looking for simplest solution
– break the task to small pieces that add value
– work incrementally
• Focus on Simple Examples of User Scenarios

Specifying Collaboratively – DoD
How will the user use the
solution? (examples!)
How can we demonstrate it?
How will we test it?
Definition of Done!

Automating the examples
• Test should be as close as possible to the DoD
– Certain tools (e.g. FitNesse) makes it easy!
• Test must fail before the code is developed…

Developing the User Story
 Develop the fixtures first (follow the errors)
 Write the minimal code to pass the tests
 Don’t add functionality that wasn’t requested!
 Apply TDD/Unit tests
 During and After development run all tests to make sure
nothing is broken
 Refactor as needed (remove duplication, give meaningful
names, etc.)
 Emergent design
 Make sure again that nothing has broken

Running the tests
 Locally, before check-in
 Automatically (CI, Nightly)
 Provide transparency!

Living documentation
Automatic Specifications = Living Documentation!
Benefits:
• A single, central source of truth
• Provides visibility about the state of the project
• Easy to identify contradicting requirements

Living documentation
Automatic Specifications = Living Documentation!
How?
• Maintain the scenarios (+some explanation text) in a
shared location
• Organize in a logical manner
– Reorganize as needed

Benefits of Specification by Example
• Full automated test coverage
– Makes refactoring easy and safe!
• Helps keeping the code clean all the time!
– Regressions are caught by the developer before check-in!
• Virtually no bugs!
• Tests are easy to understand and maintain
– Provide a comprehensive documentation which is always up to
date!
– When conflicting requirements are caught (during development),
it’s easy to communicate it to the business people

Benefits of Specification by Example
• Trust is built among stakeholders
• Forming an ubiquitous language
• Less rework – eliminate waste
• Facilitate change
– Short delivery cycles
• Receive feedback earlier!
• Respond to new requirements quickly
• Delivering the RIGHT Software!

Real Example: Promotion Engine (FitNesse)

Game – Simulating SbE

Existing application – specifications:
• Purpose: Point-of-Sale application
• Can sell products from catalog
• 2 Screens:
– Selling screen
– Products catalog

• Products catalog screen:
• Allows to create, delete and update products in the catalog
• Each product has:
• Name
• Is it sold it units or by weight?
• Price per unit or Kg
• Validations:
• Name is unique and non-empty
• Price > 0

• Selling screen:
• Only one selling at a time (no history)
• Can add, edit and remove lines
• Each line contains:
• Product name
• Quantity
• For unit products – must be an integer
• For weighable products – can be decimal
• Price – calculated by Product’s price * quantity

• Selling screen (cont.)
• A product can appear in more than 1 line
• The total is displayed at the bottom
• “Next Customer” button:
• Enabled only there’s at least 1 line
• Displays “Thank you for buying from us!”
• Cleans the screen to prepare for next customer
• “Edit Products” button:
• Enabled only when the selling is empty
• Opens the Products Catalog screen

New Requirement:
• Support Promotions from the following type:
– Given Bamba costs 4 NIS
– Promotion: “Buy 3 Bamba in 10 NIS”

The Game Flow (phase 1)
• We’ll split into 3 groups:
– Business Analysts
– Developers
– Testers
• Instructor represents the customer
• Each team will have 5 minutes to complete their role
• The other groups are waiting outside the classroom

• Business Analysts role:
– Write a specification document for the requirement
• according to their understanding of the customer’s requirement
• Can ask the Customer clarification questions
– Provide the specification document to the developers and testers

• Developers role
– Read the Specifications written by the B.A.
– Write a detailed flowchart or pseudo-code as needed
– Specify all the changes that have to be done to the app

• Testers role
– Write a test plan according to the Spec the B.A wrote
– Each test should contain clear steps and expected results
– The goal of the test plan is to find as many bugs as possible in
the Developer’s “code”

• Final stage:
– The instructor will “execute” the tests on the “Code”
• Will write which tests passed and which failed
– The instructor (as the “Customer”) will perform additional tests
and will indicate number of bugs and severity
– Note: In case we won’t find any serious bugs, we’ll repeat the
game with more complex requirements
• Phase 2 of the game will be explained later…

SbE Game – Phase 2

Simulation of Specification by Example:
• All groups work together
• Specify using examples
• Can ask the Customer for clarifications

Understanding
Specification by
Example

Specification by Example and Automatic
Testing

Manual Acceptance Tests
• Tests that are conducted by the Customer or Business
Analyst to verify that a new requested feature or User
Story is implemented as expected
• Usually done thought the UI

Manual Regression Tests
• Manual tests, (usually conducted by a tester) to verify
that old functionality still works
• Usually done though the UI

Test Case (Scenario)
• A test case represents a single scenario
• A scenario is composed of steps that interact with the
system and verifies its behavior

Test Case Structure
Arrange Act Assert
Given When Then

Test Case - Example
• Create an Item called “Bamba” with price $3
• Begin a selling transaction
• Sell 2 “Bamba”
• Ensure that the total is $6
• Pay by cash
• Ensure that the transaction is closed and its balance is 0

Automatic Acceptance Test
• An automatic test that verifies that a new requested
feature or User Story is implemented as expected
• After the test passes once, it becomes an automatic
Regression test
• Automatic UI Testing has high maintenance costs…

Test Scope
• Defintion1: The parts of the system and their
configurations that are tested by a particular test or
group of tests
• Definition2: Everything that is not under the control of the
test

Test Scope
End-to-End
UI
View Model
Client Logic
Server Proxy
Service Layer
Business Logic
DAL
ORM
DB

Test Scope
UI
View Model
Client Logic
Server Proxy
Service Layer
Business Logic
DAL
ORM
DB
Usability, UX

Types of Tests
UI
View Model
Client Logic
Server Proxy
Service Layer
Business Logic
DAL
ORM
DB
Integration

Test Scope
UI
View Model
Client Logic
Server Proxy
Service Layer
Business Logic
DAL
ORM
DB
Functional
Mock
Mock
Mock

Test Scope
UI
View Model
Client Logic
Server Proxy
Service Layer
DAL
ORM
DB
Unit tests
Business Logic

Test Scope
• Smaller Scopes ensures that the tested functionality is
handled only inside this scope and nowhere else
(loosely-coupled, lower maintenance costs).
– It aids diagnosing the root cause of problems
– Faster
• Larger scope verifies that the system behaves correctly
as a whole

Writing Good Automatic Acceptance Tests
• Create an abstraction between the test case and the
Scope.
– The Test Layer reflects the scenario in the most readable way.
• The Test Layer is agnostic to the Scope of the Test
– The Automation Layer is an adaptor between the Test Layer
and the SUT
• The Automation Layer is bound to the Scope of the Test

Automatic Tests Architecture
Automatic Test Case
(Test Layer)
Abstraction Layer
Automation Layer
UI
View Model
Client Logic
Server Proxy
Service Layer
Business Logic
DAL
ORM
DB

Choosing the right scope
• Some scopes are easier to automate than others
• Some scopes are more likely to change than others
– Causing higher maintenance costs of test code

Complementary Testing Strategies
• Manual Acceptance Tests
• Exploratory tests
• TDD / Unit tests
• Usability tests
• Performance tests
• Load and scalability tests

Deriving the Solution from the Goal

• “If I had asked my customers what they
wanted they would have said a faster horse.”
– Henry Ford

The F-16 Story

Deriving the Solution from the Goals
• Collaboration empowers creativity and simplicity
• Ask “Why?” and “Who?”
– Dig deep until you reach the “money” answer
• Focus on the outputs first, only then on the inputs
• If you’re given a solution, ask:
– How it would be useful?
– How will this solution help you achieve the goal better than what
you’re doing now?

• Benefits of Collaboration:
– Encourages Creativity
– Prevents blind spots
– Creates shared understanding
– Promotes ubiquitous language
– Produces specifications that are easy to understand
– Produced tests that are easy to maintain

Tips for effective collaboration
• Consider having introductory meetings
• Make sure that everyone have the necessary
background
• Make sure that participants have the business answers
– Involve the relevant stakeholders
– Product owner to get as much information as possible before
hand
– Don’t over-prepare!

The Power of Examples

Game: The Power of Examples

Illustrating using examples
• Examples is probably the best way to ensure
understanding
• Examples should be precise – but not over-detailed
– Focus on the relevant domain concepts and Business Rules
– Avoid technical details like specific UI details
• Expected result should be specific and measurable as
possible

Illustrating using examples
• Examples should be realistic
– But can be simplified to focus on the point at hand and be easy to
understand
• Examples and expected results should be described using the
user’s perspective
– Avoid relying on database or logs (unless the logs are aimed for the
user)
• Add a meaningful title to an example or a set of example and
a short description
• Tip: start with basic examples and explore what’s missing
• Given-When-Then!

Illustrating Non-Functional Requirements
• Many “nonfunctional” requirements can be expressed in
deterministic examples and measurements
– E.g. performance…
• Qualitative non measurable requirements (e.g. usability)
can be expressed manually using examples of “good”
and “bad”
• Use UI sketches
• Use check-lists for cross-cutting concerns

Writing Good Executable Examples
• Use Business Domain language
• Describe from User’s perspective
– Think about different roles/personas
• Focus on the relevant details!
– Hide irrelevant details in the automation layer
– Focus on the Business Logic, not on the UI

Writing Good Executable Examples
• Start from the Happy-Path and explore
• Avoid having dependencies on preconfigured,
predefined, shared data
• Have automatic tests run in an isolated environment

Game: Improving Executable Examples

Improving Executable Examples – Solution
Name: Promotion Can Add Points To Customer
Description: A promotion can add points to a customer (which he can use later in
other promotions). The user can examine his points balance in the web-site.
Steps:
• Define item: bamba (price 3NIS)
• Define promotion: For every 2 bamba – add 200 points to the customer account
• Create new customer “Arnon”, with stating balance of 100
• Begin selling for customer “Arnon”
• Add: 5 bamba
• Pay total
• Check receipt shows added points: 400
• Check balance for customer “Arnon” is 500 (though web-site)

Automating the Examples to match the
Specifications

Automating the Examples to match the
Specifications
• Choose the Tool appropriate for you
– Features and ease-of-use
– Price
– Programming Language
• Choose the automation scope

FitNesse

Concordion

Cucumber, JBehave, SpecFlow

Twist

MS-Test, NUnit

Maximizing the Effectiveness of Executable
Specifications

Validating Frequently
• Run the tests automatically
• Run Frequently
– Most Preferred: Continuous Integration
– Nightly
– Weekly
– Iteration
– Least Preferred: Release

Validating Frequently
• Separate short and long tests
• Allow developers to run the tests locally before check-in
• Optimize tests
• Run tests in parallel (on different
machines) to reduce time

Reliable Automation
• Same Code Quality as Production Code!
• Fix test issues quickly!
• Use isolated environment
• Ban dependencies between tests
• Avoid shared data
• Avoid relying on time
– Rely on events rather than timeouts
– Mock system time

Provide transparency
• Provide High-Level Report to management
• Provide detailed report for failing tests for Developers
– Make it easy to find the root cause
• Mark failing tests for incomplete user stories differently
from regression failures

Provide Transparency
• Manage failing tests
– Avoid marking tests as “Ignore”
– Make tests that fail due to known bugs so that they’ll appear in
reports
• Link to other ALM artifacts

Evolving a Documentation System

Living Documentation should be Easy to
Understand
• Executable Examples should be:
– Readable and clear for Business People
– Focused
• Provide some textual/graphical explanations
as needed
• Evolve the Language and “Refactor”
to make it consistent

Organizing the Executable Specifications
• Specifications should be easy to find
• Create links between tests/concepts (Wiki)
• Have a primary hierarchical structured according to
functional areas
– Evolve it as needed
• Use Tags for easy searching
• Have secondary indexes for other categorizations

Changing the Process

General Advice…
• Know the benefits of the new process
• Know the benefits for each stakeholder
– But also their concerns
• Listen! Listen! Listen!
– Get to know your organization and business
• Have a clear vision!
• Be enthusiastic and catch others!
• Once you started – be determined!

Benefits for Business Analysts
• Developers will actually read and understand the
specifications
– Developers won’t skip parts of the specifications
• You can track development progress easily
• You can easily identify conflicts in business rules

Challenges for Business Analysts
• I don’t do tests – that’s not my job
• It’s extra work
• They will only look at the tests and won’t read the
requirements
• What if I leave something out?
• I can’t cover all the specifications with Examples

Benefits for Testers
• You’ll be involved from the start
• You’ll build quality in from the start
• You’ll have more influence of the developed product
• You will have much better understanding of the domain
• You’ll delegate a lot of dull work to developers
• You’ll be more free for exploratory testing
• Better relationships with the Developers

Challenges for Testers
• Will I loose my job because of automation?
• Automated Tests won’t catch all problems
• I don’t want to loose control over testing
• It’s cheating – developers will know in advance what I’m
going to test…

Benefits for Developers
• Functional gaps will be flushed out before development
• Business Analysts will really understand special cases
• Automated tests will be your target for development
• You’ll have a clear Definition of Done
• You’ll be able to give better estimations

Challenges for Developers
• I don’t have time to write fixtures or maintain tests
• Acceptance Tests turn out to be incomplete
• Can’t we just write the tests ourselves?
• We already write unit tests

Tips for implementing SbE
• Ride on a wider process change (if applicable)
• Start small and show results
• Focus on improving quality
• Focus on reducing TTM
• Engage management
• Start from test automation and improve
• Create a small “Sanity” automation suite to show
benefits quickly

Tips for changing the process
• Avoid testing terminology while talking to non-testers
• Don’t focus on the tool
– Introduce a free tool instead of an expensive one…
– Avoid fighting on the right tool
• Avoid buzzwords and “Agile” terminology
• Leverage transparency in order to enforce
• Conduct effective retrospectives to learn how to improve!
• Listen carefully to what people tell you!

Conclusion

Conclusion
• What did you learn today?
• What will you take with you from this workshop?
• Make a commitment!
• Verifying the Definition of Done

THANK YOU!

Resources
• Gojko Adzic, Bridging the Communications Gap:
Specification by Example and Agile Acceptance Testing
(Neuri, 2009)
• My blog:
• Gojko Adzic, Specification By Example: How successful
teams deliver the right software (Manning, 2011)
• My blog: http://blogs.microsoft.co.il/blogs/arnona

Software quality - no more bugs!

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (19)

Similar to Software quality - no more bugs!

Similar to Software quality - no more bugs! (20)

More from Arnon Axelrod

More from Arnon Axelrod (11)

Recently uploaded

Recently uploaded (20)

Software quality - no more bugs!

Editor's Notes