Let's explore what is agile testing, how agile testing is different than traditional testing. What practices team has to adopt to have parallel testing and how to create your own test automation framework. Test automation frameworks using cucumber, selenium, junit, nunit, rspec, coded UI etc.
1. Naveen Kumar Singh
Agile Testing and Test Automation
+91 9810547500
naveenhome@gmail.com
Twitter - @naveenhome
Skype – naveen75home
2. Capacity Building Workshop by Me…
Scrum Developer Workshop
Product Discovery and Requirement Analysis
Test Driven Development
Behavior Driven Development
Agile Testing and Test Automation
Agile and Scrum Foundation
DevOps Foundation
Continuous Integration and Delivery
Agile Coaching and Facilitation
Agile Technical Practices
3. Program Overview
• This workshop focuses primarily on agile testing techniques and
processes in addition to the mindset and role of an agile tester.
• Learning outcomes include the ability to distinguish different
types of testing on an agile project.
• Understanding how business, development, and testing
personnel best collaborate on an agile development cadence.
4. Learning Objective
• What is Agile and available agile frameworks
• What is Agile Testing?
• Agile mindset and culture for continuous testing
• Agile Testing Quadrants and category
• Test Automation Pyramid
• Pair Programming, Pair Testing and Peer review
• Example Driven Development – Spec by Examples, ATDD, BDD
• Feature, story, unit and NFRs testing
• Roles and Responsibilities of Test Engineers and Managers
• Test Strategies and planning
• Agile testing metrics
• Successful delivery
• Test Environment and Infrastructure
• Working on Distributed Team
5. Risk in Software Development
Schedule slips
Project canceled
System goes sour
Defect rate
Business misunderstood
Business change
False feature rich
Staff turnover
6. Agile Manifesto
We are uncovering better ways of developing software by doing
it and helping others do it. Through this work we have come to
value
Individuals and
Interactions
Processes and Toolsover
Working Product
Comprehensive
Documentation
over
Customer Collaboration Contract Negotiationover
Responding to Change Following a Planover
That is, while there is value in the items on the right, we value
the items on the left more
7. Agile Principles
I. Our highest priority is to satisfy the customer through early and continuous delivery of
valuable software.
II. Welcome changing requirements, even late in development. Agile processes harness
change for the customer's competitive advantage
III. Deliver working software frequently, from a couple of weeks to a couple of months, with
a preference to the shorter timescale
IV. Business people and developers must work together daily throughout the project.
V. Build projects around motivated individuals. Give them the environment and support they
need, and trust them to get the job done.
VI. The most efficient and effective method of conveying information to and within a
development team is face-to-face conversation.
VII. Working software is the primary measure of progress.
VIII.Agile processes promote sustainable development. The sponsors, developers, and users
should be able to maintain a constant pace indefinitely.
IX. Continuous attention to technical excellence and good design enhances agility.
X. Simplicity--the art of maximizing the amount of work not done--is essential.
XI. The best architectures, requirements, and designs emerge from self-organizing teams.
XII. At regular intervals, the team reflects on how to become more effective, then tunes and
adjusts its behaviour accordingly.
10. How to select right process framework?
Source - Wikipedia
11. Agile Testing
Building quality into the product rather than trying to test it in
later.
How agile testing is different from traditional approach if team
still struggling to answer what testers will do at beginning of
sprint?
13. Found a bug during deployment?
Manager/
Customer
Testers Developer
Architect Business
Analyst
Product
Manager
14. Agile Testing
Let’s understand by a simulation game to understand why
agile testing is important?
Simulation will cover below:-
• Uncovering problem
• Whole Team Approach
• Agile Testing Principles
15. Whole Team Approach
• Sub-optimization Vs Whole Team
• Care about Cost of Delay
• Calculate Cost of Production
• Remove waste from software development
16. Waste in Software Development
Defects Overhead processes Waiting
Not-realizing
people's potential
Task switching Inventory
Motion Extra features
18. Testing Approach
Plan Quality - Test Strategy, Test Data, Infrastructure
Built-In Quality - Specification By Examples, API vs GUI
Testing, Regression Testing, NFR testing, Tools to test and
provide feedback
Validate Quality - Acceptance Test Driven Development, UAT,
CAT
20. Inverted Testing Pyramid
Automated
UI Test
Automated API Test
Automated Service Test
Automated Component Test
Automated Unit Test
Exploratory
Test
70%
20%
10%
21. Brian Marick’s Testing Quadrants
Functional Tests
Story Tests
Prototypes
Simulations
Exploratory Tests
Usability Tests
User Acceptance
Tests (UAT)
Performance Testing
Load Testing
Security testing
NFR Testing
Unit Tests
Component Tests
Integration Tests
Automated
& Manual
Manual
Automated Tools
SupportingTheTeam
CritiqueProduct
Business Facing
Technology Facing
22. Pair Programing
• Two developers work together to accomplish a single task.
Driver and Navigator
• One developer focus on details of task and other focus
fitment of task in whole project
• Produces code through conversation
• Reinforce good programming habits
• May be uncomfortable at first
• Two brains on one task and two sets of eye produce fewer
bugs
• Investment in continual training
27. Building right or right product?
Business Failure
Useless Stuff
Business Success
Technical Debts
Specification By
Example
Build it Right
Build the Right Things
28. Specifications by Example
Impact Mapping
(Vision)
Story Mapping
(MVP & MMF)
User Story
(Specification)
As A Student I want to Purchase used Books Online So that I can
save money
29. What are Specifications by Example
Thin Slices of System Behavior
That Deliver Business Value
Described as concrete examples
That are potentially automatable
To create executable specifications
Captured in live documentation
30. Refining the Specifications
Specifications with examples are acceptance tests. Gojko Adzic
• Be Precise and make sure specs are testable
• Avoid “Scripts” and “Flows”
• Focus on business functionality not design
• Avoid UI details
• Avoid covering every possible combination
34. Write Specification
Write examples for one specification.
In order to maintain health record
As a patient
I want to enter my doctor visit details
35. Discussion of Acceptance Criteria
If(user==“new”
{
object = user
}
Else
error
We should encourage
employee to update
ride
Login -> Click on
“New” and enter test
data and then Result
&%$^&
36. Collaboration of 3 amigos
Technical
Feasibility
Happy Path
Exceptions, Test
Data, Boundary
Conditions
Developer Business Tester
37. Collaboration of 3 Amigos
In order to maintain health record
As a patient
I want to enter my doctor visit details
• New prescription can be uploaded as Image
• Duplicate prescription not allowed
• Photo can be captured by application
• New prescription can be added manually also
38. Collaboration of 3 Amigos
In order to maintain health record
As a patient
I want to enter my doctor visit details
• New prescription can be uploaded as Image
• Books can be removed from shopping cart
• Shopping basket should be empty for new
user
• User should not be able to add same book
twice
Given “James” want to upload new prescription
When “James” selected “DoctorVisit.jpeg”
And Click on “Record”
Then “Prescription Uploaded” displayed
39. Gherkin
Feature – Name of feature
Scenario – Behavior to be developed
Given – Pre-conditions
When – Actions to be performed
Then – Expected Result
And – Use for multiple Given, When & Then
But – Describe exception cases
Scenario Outline – Define multiple scenarios
Examples – Multiple Scenarios
Background – Avoid repeated Given
42. Let’s get started
Hands-on Behavior Driven Development
• Write Feature to describe scenario in Gherkin
• Eclipse, Java, Cucumber, Selenium, Junit
• Create Test Runner class file
• Generate Steps file
• Write code to pass test
43. BDD - Characteristics
• A testable story (it should be the smallest unit that
fits in an iteration)
• The title should describe an activity
• The narrative should include a role, a feature, and a
benefit
• The scenario title should say what's different
• The scenario should be described in terms of Givens,
Events, and Outcomes
• The givens should define all of, and no more than,
the required context
• The event should describe the feature
44. BDD – Feature File
Feature: Checkout
In order to calculate price of groceries
As a Store Staff
I should be able to calculate price for groceries during
checkout
Scenario: Checkout a banana
Given The price of a “Banana” is 5
When I checkout 1 “Banana”
Then the total price should be 5
45. BDD – Test Runner Class
package test.java;
import org.junit.runner.RunWith;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
@RunWith(Cucumber.class)
@CucumberOptions(features = "src/test/resources")
public class testrunner {
}
46. BDD – Run Output
1 Scenarios ([33m1 undefined[0m)
3 Steps ([33m3 undefined[0m)
0m0.000s
You can implement missing steps with the snippets below:
@Given("^The price of a "([^"]*)" is $(d+)$")
public void The_price_of_a_is_$(String arg1, int arg2) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
@When("^I checkout (d+) "([^"]*)"$")
public void I_checkout(int arg1, String arg2) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
@Then("^the total price should be $(d+)$")
public void the_total_price_should_be_$(int arg1) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException(); }
47. BDD – Step File
public class TestSteps {
@Given("^The price of a "([^"]*)" is $(d+)$")
public void The_price_of_a_is_$(String arg1, int arg2) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
@When("^I checkout (d+) "([^"]*)"$")
public void I_checkout(int arg1, String arg2) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}
@Then("^the total price should be $(d+)$")
public void the_total_price_should_be_$(int arg1) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
} }
48. BDD – Modified Step File
public class TestSteps extends TestCase {
String product;
int rate, qty;
@Given("^The price of a "([^"]*)" is $(d+)$")
public void The_price_of_a_is_$(String arg1, int arg2) throws Throwable {
product = arg1;
rate = arg2;
}
@When("^I checkout (d+) "([^"]*)"$")
public void I_checkout(int arg1, String arg2) throws Throwable {
qty=arg1;
}
@Then("^the total price should be $(d+)$")
public void the_total_price_should_be_$(int arg1) throws Throwable {
if(product.equals("Banana"))
assertEquals(arg1, rate*qty); }}
50. BDD – Lifecycle
Your Project Features Scenarios Steps
Your System
Automation
Library
Support Code
Step
Definitions
Technology
Facing
Business
Facing
51. Gherkin – Background
Feature: Feedback when entering invalid credit card
details
In user testing we've seen a lot of people who made
mistakes entering their credit card. We need to be as
helpful as possible here to avoid losing users at this
crucial stage of the transaction.
Background:
Given I have chosen some items to buy
And I am about to enter my credit card details
52. Gherkin – Background
Scenario: Credit card number too short
When I enter a card number that's only 15 digits long
And all the other details are correct
And I submit the form
Then the form should be redisplayed
And I should see a message advising me of the correct
number of digits
53. Gherkin – Multiple AND
Scenario: Expiry date invalid
When I enter a card expiry date that's in the past
And all the other details are correct
And I submit the form
Then the form should be redisplayed
And I should see a message telling me the expiry date
must be wrong
54. Gherkin – Comments
# This feature covers the account transaction and
hardware-driver modules
Feature: Withdraw Cash
In order to buy beer
As an account holder
I want to withdraw cash from the ATM
# Can't figure out how to integrate with magic wand
interface
Scenario: Withdraw too much from an account in
credit
Given I have $50 in my account
55. Gherkin – Step Definitions
In Cucumber, results are a little more sophisticated
than a simple pass or fail. A scenario that’s been
executed can end up in any of the following states:
• Failed
• Pending
• Undefined
• Skipped
• Passed
These states are designed to help indicate the
progress that you make as you
develop your tests.
56. Gherkin – Test Runner Setup
Element Purpose Default
dryRun true (Skip execution of glue code) False
strict true (will fail execution if there are undefined or
pending steps)
False
glue where to look for glue code (stepdefs and hooks) {}
features the paths to the feature(s) False
monochrome whether or not to use monochrome output False
format what formatter(s) to use {}
tags hat tags in the features should be executed {}
58. Acceptance-TDD
• What is A-TDD?
• How A-TDD is different than BDD
• Inside – out Vs Outside – in
• Tools for ATDD
• How FitNesse is different than Cucumber
59. Let’s build software using A-TDD
Let’s try FitNesse
• Setup FitNesse
• Write tests on WIKI
• Create Fixture
• Develop system under test
• Execute Test from Wiki
60. Test Driven Development
• Unit test
• Why TDD
• How TDD is different than A-TDD
• What is Right-BICEP
• Mocking
• Available tools
• TDD vs A-TDD vs BDD
61. Unit Testing
• A unit test is to test one unit of work. Generally
one unit means of one requirement for one
method.
• Unit testing functional testing
• Unit testing is not also User Interface testing
62. Unit Testing - Characteristic
Isolated from the other code
Isolated from the other developers
Targeted
Repeatable
Predictable
63. Test Driven Development
Let’s try to develop a component using TDD
focusing Right-BICEP
Expected result must be RIGHT
B – Boundary Condition
I – Inverse Check
C – Cross check
E – Error Condition
P – Performance condition
64. Boundary Condition
• Totally bogus or inconsistent input values, such
as a file name of "!*W:Xn&Gi/w>g/h#WQ@".
• Badly formatted data, such as an e-mail address
without a top-level domain ("fred@foobar.").
• Empty or missing values (such as 0, 0:0, "", or
null).
• Values far in excess of reasonable expectations,
such as a person's age of 10,000 years.
65. Boundary Condition
• Duplicates in lists that shouldn't have
duplicates.
• Ordered lists that aren't, and vice-versa. Try
handing a pre-sorted list to a sort algorithm, for
instance - or even a reverse-sorted list.
• Things that arrive out of order, or happen out of
expected order, such as trying to print a
document before logging in, for instance.
66. Boundary Condition - CORRECT
Conformance – Does the value conform to an
expected format?
Ordering – Is the set of values ordered or
unordered?
Range – Is the value within reasonable range?
Reference – Does the code reference anything
external that isn’t under direct control of code?
Existence – Does the value exist? (not-null etc.)
Cardinality – Are there exactly enough values?
Time (absolute and relative) – everything happening
at right time?
67. Test Driven Development
TDD was introduced long back in 1999 as part of
Extreme Programming (XP) by a group of developers.
Basic philosophy is “Before you write code, think
about what it will do”.
Testing code once will not ensure future changes will
not impact to earlier tested code so TDD is something
you write once but run once, twice, three times etc.
TDD is risk averse programming, investing work in
the near term to avoid failures later on.
68. Test Driven Development
When TDD used as an application design
methodology, it works best when the business user is
engaged in the process to help the developer define
the logic that is being created. Even possible to define
a set of input and expected output.
TDD ensures quality code from the start. Write only
the code needed to pass the test so method is going to
have less code. Lesser the code lesser opportunity for
defects.
69. Test Driven Development
TDD ensures a high degree of conformity of the
business requirement
TDD helps keep unused code out of the system
TDD provides built-In regression testing
71. Test Driven Development - Rules
First Law You may not write production code until
you have written a failing unit test.
Second Law You may not write more of a unit test
than is sufficient to fail, and not compiling is failing.
Third Law You may not write more production code
than is sufficient to pass the currently failing test.
72. Misconceptions
• Writing code 1st then Test is same like TDD
• TDD isn’t useful for designing architecture
• Takes too much time
• Too complex to learn and implement
• TDD is only good for small project
73. Test Double
Test double is just like replacing on-screen Actor with
stunt actor during film making.
4 ways to achieve test double
• Dummy
• Stub
• Mock
• Fake
74. Refactoring
• Refactoring is the act of changing the internal
implementation of a class or method with the
aim of making the code more readable and
maintainable.
• Refactoring also reduces the code’s overall
complexity without changing the external
behavior of the class or method.
• Refactoring allows you to continuously change
and improve your code.
75. Code Smells
Code smells are simply a collection of commonly
known and widely found code-based anti-patterns.
77. Technical Debts
Also known as design debt or code debt is "a
concept in programming that reflects the extra
development work that arises when code that is easy
to implement in the short run is used instead of
applying the best overall solution".
Technical debt can be compared to monetary debt. If
technical debt is not repaid, it can accumulate
'interest', making it harder to implement changes
later on. - Wikipedia
82. When To Refactor
The Rule of Three: The first time you do
something, you just do it. The second time you do
something similar, you wince at the duplication,
but you do the duplicate thing anyway. The third
time you do something similar, you refactor.
When add new function/method
When you need to fix
During code review
83. Exploratory Testing
• How is different than other testing?
• Concurrent test design and execution
• Balancing it with automated test
• Why to perform exploratory testing?
• Does exploratory testing add value?
• Do we need to plan for exploratory testing?
84. Non-Functional Requirement
• Need for Non-Functional Requirement Testing
• What are the area needs to cover as part of NFR
• Isn’t it NFR part of customer requirement?
• Different type of NFR testing such performance,
security and compliances etc.
85. Test Strategies
• Strategy is a thought process not a documents
• Preventive vs Corrective
• Team is responsible not just test engineer
• Developer test vs Tester test
• Business centric vs Technology centric
• Based on – Goal, Risk, Opportunity and Constraints
87. Test Metrics
• Test pass/fail rates
• Defects discovery rate
• Regression test results
• Defects density
• Found and fixed cycle time
• Code/test coverage
• Requirement coverage
It’s a trap! Don’t fall for it. Focus on value delivery.
88. Environment and Infrastructure
Test Environment
Configuration Management
Automated Build
Continuous Integration
Continuous Delivery
Delivery Pipelines
Test Data Management
89. Continuous Integration
A software development practice where members of
the integrate their work frequently, usually each
person integrates at least daily- leading to multiple
integrations per day. Each integration is verified by an
automated build to detect integration errors as
quickly as possible. Many teams find that this
approach leads to significantly reduced integration
problems and allows a team to develop cohesive
software more rapidly – Martin Fowler
91. Continuous Integration
• All developers run private builds on their own
workstations before committing their code
• Developers commit their code to a version control
repository at least once a day.
• Integration builds occur several times a day on a
separate build machine.
• 100% of tests must pass for every build.
92. Source Code Management
This is where developers commit their changes.
Purpose of version control repository is to manage
changes to source code and other software assets
(docs, and database scripts).
Popular Version Control – SVN, TFS, CVS, GIT
93. CI Server
A CI server runs an integration build wherever a
change is committed to the version control
repository. Usually CI server is configured to check
version control repository every few minutes.
The CI servers will pull out latest changes and run
build scripts to produce new build/product.
94. Continuous Build but why?
Improve Product Quality
Save time and cost
Eliminate redundant task
Minimize “Bad Builds”
Eliminate dependency on individual
Need for continuous integration
95. Continuous Build
• Manual Build Vs Automated Build
• Automation using build tools vs Script
• Available Build Tools
• Server Type
• On-demand
• Scheduled
• Triggered
• Continuous Build using Maven/MS Build/Rake/Gradle
96. Build Script
• Build script is a single script or set of scripts to compile,
test, inspect and deploy software.
• Build script can be used without CI system. Ant, NAnt,
MSBuild and Rake etc are example of build tools.
• Team usually build through IDEs.
Continuous Integration is much more than just
compilation of code.
97. Continuous Testing and Inspection
Many consider CI without automated test is not useful. This is
not true because you can still execute many other types of
testing load, component and security etc.
Automated code inspections can be used to enhance the
quality of the software by enforcing rules.
You can use CI system to run these automatically against a
code base. Tools like FXCOP and Stylecop is popular for
Microsoft platform. SonarQube, CheckStyle, JavaNCSS and
CPD are for Java platform.