Many projects implicitly use some kind of risk-based approach for prioritizing testing activities. However, critical testing decisions should be based on a product risk assessment process using key business drivers as its foundation. For agile projects, this assessment should be both thorough and lightweight. PRISMA (PRoduct RISk MAnagement) is a highly practical method for performing systematic product risk assessments. Learn how to employ PRISMA techniques in agile projects using risk-poker. Carry out risk identification and analysis, see how to use the outcome to select the best test approach, and learn how to transform the result into an agile one page sprint test plan. Practical experiences are shared and results achieved employing product risk assessments. Learn how to optimize your test effort by including product risk assessment in your agile testing practices.

1. W10
Agile Testing
5/7/2014 1:45:00 PM
Risk-Based Testing for Agile
Projects
Presented by:
Erik van Veenendaal
Improve Quality IT Services BV
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Erik van Veenendaal
Improve Quality IT Services BV
A leading international consultant, trainer, and recognized expert in software testing, Erik van
Veenendaal (erikvanveenendaal.nl) is the founder of Improve Quality Services BV, a company
that specializes in testing, requirements engineering, and quality management. Erik is the
author of a number of books and papers, a core developer of the TMap testing methodology
and the TMMi improvement model, a participant in the International Requirements Engineering
Board working parties, currently on the TMMi Foundation board, and a frequent speaker at
international testing conferences. For his major contribution to the field of testing, Erik received
the 2007 European Testing Excellence Award.

3. © 2011 Improve Quality Services BV PRISMA tutorial- 1
1
Practical Risk-Based Testing
also for
Agile projects
PRISMA®
Erik van Veenendaal
www.erikvanveendaal.nl
Improve Quality Services BV 2
Erik van VeenendaalErik van Veenendaal
Founder and major shareholder ImproveQS
In testing since 1989 working for many
different clients and in many different roles
Author “TMap”, “TMMi model” and many
other books and papers
Former Vice-President International Software
Testing Qualifications Board (ISTQB)
Supporting member IREB board
Keynote speaker, e.g. EuroSTAR, STAR
Winner of the European Testing Excellence
Award
www. erikvanveenendaal.nl

4. © 2011 Improve Quality Services BV PRISMA tutorial- 2
Improve Quality Services B.V. 3
Testing ChallengesTesting Challenges
Increasing importance and size of
software in society as a whole
− e.g., amount of software in consumer product
doubles every 24 months,
− number of defects is almost constant
High Competition & Outsourcing
− Time-To-Market, Product Quality, Price Levels are essential for
business success
Testing often takes 30 - 40% of project costs
Risk based testing deals with both effectiveness
and efficiency
Improve Quality Services B.V. 4
What is Risk?What is Risk?
“A factor that could result in a future negative
consequence; usually expressed as impact and
likelihood” (ISTQB Glossary)
Testers ‘only’ have the responsibility to identify the
risks and provide information on their status
“to dare to undertake”
− management attitude and style ..

5. © 2011 Improve Quality Services BV PRISMA tutorial- 3
Testing = Risk ManagementTesting = Risk Management
Objective: most feasible coverage
− effective usage of limited resources
− optimize test effort
Limited resources
staffing
infrastructure
time !
..
The right level and type of coverage on the right
parts at the right time
Always applicable, whether it is V-model, Agile or ....
Improve Quality Services B.V. 5
Our Challenge .Our Challenge .
Improve Quality Services B.V. 6
ifif only weonly we
knew !!knew !!

6. © 2011 Improve Quality Services BV PRISMA tutorial- 4
RiskRisk--Based TestingBased Testing
Risk identification looks at ways of
establishing what the risks are and where
they are
Risk analysis looks into the critical, complex
and potential error prone areas
Then we build tests to mitigate the risk
Subsequently we monitor and report
regarding the risks
Improve Quality Services B.V. 7
Based on Practical ExperiencesBased on Practical Experiences
Improve Quality Services B.V. 8
8

7. © 2011 Improve Quality Services BV PRISMA tutorial- 5
Risk IdentificationRisk Identification
Split up test basis in functional or technical items
V-model: based on requirements
Agile: backlog items / user stories for next sprint
Can also be based on a brainstorm session
Maximum number of appr. 35 risk items
Improve Quality Services B.V. 9
Risk item 1 Functionality
Risk item 2 Security
Risk item 3 Functionality
Risk item 4 Interoperability
RiskRisk AnalysisAnalysis
Risk = impact x likelihood
− What is the impact for the business ?
− What is the likelihood that there are defects ?
Determine factors based on previous projects, e.g.,
defect patterns
Improve Quality Services B.V. 10
Impact – business risk
Likelihood
technicalrisk
You already know this !
Exercise: Risk Factors

8. © 2011 Improve Quality Services BV PRISMA tutorial- 6
Improve Quality Services B.V. 11
Factors From PracticeFactors From Practice
Likelihood
− complexity
− new development
(level of re-uses)
− interfacing
− size
− technology
− geographical spread
− inexperience
(of development team)
Impact
− business importance
(“selling item”)
− financial (or other)
damage (e.g. safety)
− usage intensity
− external visibility
− cost of rework
Customization
required
defect patterns / history
Risk Assessment in a NutshellRisk Assessment in a Nutshell
1) Stakeholder analysis: Assigning Factors
2) Individual scoring
3) Consensus meeting
4) Position the risk items in a Product Risk Matrix
5) Define a differentiated risk-based test approach
Improve Quality Services B.V. 12
Business
importance
Usage
intensity
Safety
Item 1
Item 2
Item 3
Item 4
5
5
4
5
5
4
3
2
they shall
make
choices

9. © 2011 Improve Quality Services BV PRISMA tutorial- 7
Product Risk MatrixProduct Risk Matrix
Improve Quality Services B.V. 13
IIII
IVIV
II
IIIIII
L
H
M
L HM
x
x
x
x
x
Likelihood
Impact
Must Test
Should Test
Could Test
“Won’t Test”
focus of
unit testing
focus of
acceptance
testing
MoSCoW priorities
AgileAgile -- Play the Card Game:Play the Card Game: RiskRisk--PokerPoker
Poker Planning / Wide Band Delphi based
Improve Quality Services BV 14

10. © 2011 Improve Quality Services BV PRISMA tutorial- 8
Improve Quality Services B.V. 15
Wide Band Delphi / Poker Planning (1)Wide Band Delphi / Poker Planning (1)
Bottom-up estimation techniques
- wide variance in estimates
- commitment of people involved
- accurate estimation we use the collective wisdom of the team
Each estimator has cards with valid estimates
A user story (product backlog item) is discussed
Each estimator selects an estimate
Cards are turned over
Discuss differences (especially outliers)
Repeat until consensus
Improve Quality Services B.V. 16
User Story
Explain
Estimates
Collect
Share info
OK?
Final
estimate
Moderator Estimator
OK?
Wide Band Delphi / Poker Planning (2)Wide Band Delphi / Poker Planning (2)

11. © 2011 Improve Quality Services BV PRISMA tutorial- 9
Improve Quality Services B.V. 17
Risk Poker the same, butRisk Poker the same, but
Scoring separately for likelihood and impact
Discussion shall be factor based
Limited number of cards for each color (value)
Agreement, Product owner (impact) and lead-
developer (likelihood) act as tie breaker
Improve Quality Services B.V. 18
Example User Story
As as bank account holder, I want a yearly financial
overview of my accounts, so that I can submit my taxes
Impact ?
Likelihood ?
Let’s PlayLet’s Play
Impact:
Likelihood:

12. © 2011 Improve Quality Services BV PRISMA tutorial- 10
Improve Quality Services B.V. 19
AgileAgile PRISMAPRISMA RiskRisk MatrixMatrix
See any “problems” with this?
Example
User
Story
Improve Quality Services BV 20
Differentiated Test Approach !!Differentiated Test Approach !!
Test design
techniques
Reviews & inspection
Test design start-up
meetings
Reviews of test design
Level of detail of test
cases
Exit criteria
Level of independence
More time & effort
Most experienced
person
Priority setting
Regression testing
Re-testing
without this risk assessment
doesn’t make much sense !!

13. © 2011 Improve Quality Services BV PRISMA tutorial- 11
Improve Quality Services BV 21
RealReal--LifeLife AgileAgile ExampleExample
IIII
IVIV
II
IIIIII
LL
HH
MM
LL HMM
x
x
x
x
x
Likelihood
Impact
Must Test
Should Test
Could Test
“Won’t Test”
Test design techniques
Review design
Support module tests
Review module tests
Test design techniques
Review design
Exploratory Testing
Support module tests
Review module tests
Exploratory Testing
AgileAgile ““OneOne Page Test Plan”Page Test Plan”
Improve Quality Services B.V. 22
Must Test
.. Test Approach ..
Should test
Test Approach ..
Could Test
.. Test Approach ..
Would Test
.. Test Approach ..
Shall include
a definition of “done”
A 1-2 hour session
to achieve a one page
Sprint Test Plan

14. © 2011 Improve Quality Services BV PRISMA tutorial- 12
CommunicationCommunication
Improve Quality Services B.V. 23
Product Risk MatrixProduct Risk Matrix –– InitialInitial
US
58
US
31US
49
US
27
Impact - Business risk
Likelihood-TechnicalRisk
IVIII
III

15. © 2011 Improve Quality Services BV PRISMA tutorial- 13
Product Risk MatrixProduct Risk Matrix –– ProgressProgress
US
58
US
31US
49
US
27
Impact - Business risk
Likelihood-TechnicalRisk
IVIII
III
Improve Quality Services BV 26
Key learning pointsKey learning points
A structured andA structured and practical approachpractical approach forfor
riskrisk--basedbased testing istesting is availableavailable
Make itMake it AgileAgile, but use the formal rules, but use the formal rules
Efficient, Effective and AttractiveEfficient, Effective and Attractive
DefineDefine aa riskrisk--basedbased differentiated testdifferentiated test
approachapproach
ProvideProvide riskrisk--basedbased trackingtracking && reportingreporting

16. © 2011 Improve Quality Services BV PRISMA tutorial- 14
DownloadsDownloads
On www.erikvanveenendaal.nl
A full PRISMA white paper (e-book)
Paper: 10 Recommendations on
Risk-Based Testing
Paper: Product Risk Assessments in Agile Projects
(Risk-Poker)
To acquire the PRISMA freeware tool sent a
request to info@improveqs.nl
Book “Practical Risk-Based Testing; The PRISMA
Approach”, ISBN9-78490-986070 (www.utn.nl)
Improve Quality Services B.V. 27
Improve Quality Services B.V. 28