If you’ve tried to make testing really count, you know that “risk” plays a fundamental part in deciding where to direct your testing efforts and how much testing is enough. Unfortunately, project managers often do not understand or fully appreciate the test team’s view of risk—until it is too late. Is it their problem or is it ours? After spending a year on a challenging project that was set up as purely a risk mitigation exercise, George Wilkinson saw first-hand how risk management can play a vital role in providing focus for our testing activities, and how sometimes we as testers need to improve our communication of those risks to the project stakeholders. George provides a foundation for anyone who is serious about understanding risk and employing risk-based testing on projects. He describes actions and behaviors we should demonstrate to ensure the risks are understood, thus allowing us to be more effective during testing.

1. Risk-based Testing: Not for the
Fainthearted
George Wilkinson
Grove Consultants
george@grove.co.uk
www.grove.co.uk

2. Risk…a definition
Definition: ISO 31000: (2009)
Risk Management standard
definition, defines risk as:
“the effect of uncertainty
on objectives whether
positive or negative”
A more common project
definition of ‘risk’ is:
“the probability of an event,
hazard, threat or situation
occurring resulting in
undesirable consequences”

3. Testing…the challenges
risks are present due to:
a) Time pressures
b) System sophistication
c) Budget restrictions
d) Changing requirements
e) Resource challenges
…and on…and on…and on…
Test: a definition
“a particular process or method for
trying or assessing”
Our objectives:
• Find defects
• Raise confidence
• Have conformance to regulation
• Combinations of the above
In the act of
„testing‟ we need
FOCUS!!!

4. The TiNA project
London Civil and
Military Airspace
System
Existing FDP
system
Replacement FDP
system

5. Why is your system important?
flight numbers are
continually growing
world-wide
in 2010:
over the entire UK 2.1
million flights were
handled in controlled
airspace
over 200 million
passengers flew
“The safety of the
passengers rely upon
this system”
do you have a reason why
your system is important?
…if not find one!

6. Risk Exposure =
Probability * Consequence
Understanding Risk
Probability
Risk 1
100% 5
Risk Exposure = 2*2 = 4
75% 4
Risk 2
Risk Exposure = 4*2 = 8
50% 3
Risk 3
25% 2
1%
Risk Exposure = 2*4 = 8
Risk
Risk 4
1
1
2
3
4
5
Risk Exposure = 4*4 = 16
Consequence

7. Risk Management
risk
identification
•
•
•
•
•
risk workshops
expert interviews
heuristics
lessons learned
checklists
 Investig


risk
analysis
risk
mitigation
(control)
Try to…calculate
Risk Exposure


Risk
monitoring
ENGAGE THE STAKEHOLDERS!
ate
Ignore
Project
Mitigatio
n
Test
Conting
ency
plan

8. Risk-based Testing…points to note
can be hard to commit to…
can help answer:
how much testing is enough
is a risky business in itself
does not mean test all known risks
The main initiative being…
to provide detailed information to help „management‟
make a decision on whether a system is ready for
release

9. Is it them or is it us…?
Number
Software Risk item Frequency
Frequency
1
Misunderstanding of requirements
5
1
Lack of top management support
5
3
Lack of adequate user involvement
4
4
Failure to gain user commitment
3
5
Failure to manage end user expectations
3
6
Changes to requirements
3
7
Lack of an effective project management methodology
3
Project Management is project focussed
Test Management is product focussed
Source: Top Ten Lists of Software Project Risks: Evidence from Literature Survey by
Tharwon Arnuphaptrairong

10. Behaviours in Risk Based Testing #1
build the relationship with Project Management
test need to work close with Project Management
appreciate Project Management objectives
push Project Management for risk based decisions
learn to use „risk language‟
when verbalising risks
when writing risks

11. Behaviours in Risk Based Testing #2
all „risk management‟ activities should be efficient
act boldly, time is limited
ensure risk workshops are facilitated well
learn the lessons from past mistakes
retain the „risk based approach‟
in strategy and planning
in execution
on the next release
….when you are tired
Chance
of failure
Test
and report
Risk
Management
Plan
For Test

12. Risk-based reporting
start
today
end date
residual
risks of
releasing
TODAY
all risks
„open‟ at
the start
Progress through the test plan
Source: Risk Based E-Business Testing – Paul Gerrard & Neil Thompson

13. Defect data provides a message…choose well
defect density
70
60
50
40
defect density analysis
30
20
10
0
140
120
100
defects found versus fixed
80
Found
60
Fixed
40
20
0
Day Day Day Day Day Day Day Day Day Day
1
2
3
4
5
6
7
8
9
10

14. Summary
risks in modern systems are prevalent and testing
requires focus
risk based testing requires:
a good relationship with project management
an efficient risk management approach
good communication and reporting
discipline
success comes with taking
risks