Software Testing Online meetup of Open Tech Week by Sigma Software, 2020

1. Risk-based test techniques on
Agile projects
Anton Muzhailo
Quality Architect, GlobalLogic Ukraine
March 2020

2. Anton Muzhailo
QA Architect, GlobalLogic
• 10 years in Quality Assurance
• ISTQB Certified Test Manager
• IT Awards Quality Assurance Winner 2019
• 5 years of mentoring experience
About me
in/muzhailo/

3. Why testing fails on agile projects?
Priority
changes
1
Scope
changes
2
Unclear
requirements
3
And one
more thing…
4

4. Why testing fails on agile projects?
Priority
changes
1
Scope
changes
2
Unclear
requirements
3
And one
more thing…
4
You are not Agile
enough

5. Risk-based testing will help!
RisksTest
estimation
Release
readiness
Test
scope
Testing
depth
Test
reduction
Test
prioritization

6. Quick reminder
Risk
characteristics
Impact Likelihood
Risk Level = Likelihood * Impact
Technical
stakeholders
Determine
Business
stakeholders
Determine

7. Risk-based vs Req-based
Feature Acceptance
Criteria 1
Feature Acceptance
Criteria 2
Test case 1
Test case 2
Test case 3
Test case 1
Test case 2
Test case 3

8. Risk-based vs Req-based
Feature Acceptance
Criteria 1
Feature Acceptance
Criteria 2
Test case 1
Test case 2
Test case 3
Test case 1
Test case 2
Test case 3
Risk1
Risk2
Risk3

9. Risk-based testing activities
Risk
identification
Risk
assessment
Risk
mitigation
Risk
management

10. Risk Identification Techniques
Independent
assessments
Project
retrospectives
Brainstorming
Checklists
Calling on past
experience

11. Risk Assessment
Determining the likelihood and impact for each risk
Evaluating or assigning other properties of each risk,
such as risk owner
Cutting down the risks we don’t care about

12. Risk Mitigation
Testing: Static and Dynamic
Risk Contingency
Accepting the risk

13. Risk Management
Techniques
Systematic Software Testing
Pragmatic Risk Analysis and
Management
Product RISk MAnagement Hazard Analysis
Cost of exposure Failure Mode and Effect Analysis
Quality Function Deployment Fault Tree Analysis
Process Decision Program Chart Event Tree Analysis

14. Systematic Software Testing
Forming a
Brainstorming
Team
Compile a
list of features
Determine
the likelihood
Determine
the Impact
Assign
numeric
values (RL)
Compute the
risk priority
Review/modify
the values
Prioritize the
features
Determine
the cut line
Consider
mitigation
1 2 3 4 5
6 7 8 9 10

15. Movie subscription system Likelihood Impact Risk
LevelFeatures Attributes
Change resolution 0.9 0.9 0.81
To be
tested
Play the movie 0.5 1 0.5
Performance 0.25 0.9 0.225
Security 0.25 0.25 0.125
Buy subscription 0.1 1 0.1
See
recommendations
0.9 0.25 0.225
Accepted
Create account 0.1 1 0.1 Not to be
tested
(or less)
Recover account 0.1 0.9 0.09
Usability 0.25 0.25 0.625

16. Hazard Analysis
Causes Hazard Consequences
Risk
Reduction
Strategies
Mitigation
actions
Safety
Requirements
Hazard Analysis
Risk Reduction
Hazard analysis techniques
include:
• Function Failure Analysis
• Event Tree Analysis
• Failure Modes and Effects
Analysis
• Fault Tree Analysis
• Cause-consequence Diagrams
• Hazard and Operability
Studies.

17. Cost of exposure. Risk exposure
Risk exposure (RE) = 2% * 80K = 1600
Risk Probability of
occurring
Total loss if it
occurs
Risk exposure
Product recall situation 2% 80K 1600
Significant product rejection 0.1% 100K 1000
Competitive strike 10% 25K 2500
Most serious risk

18. Cost of exposure. Risk Reduction Leverage
Risk
Probability of
occurring
Total loss if it
occurs
Risk
exposure
Competitive strike 10% 25K 2500
Countermeasure Total cost New risk
probability
New total loss New Risk
exposure
RRL
Advertising campaign 40K 3% 5K 150 0.059
Price promotions 30K 5% 10K 500 0.067
Simultaneous launch 10K 8% 15K 1200 0.13
Risk Reduction Leverage (RRL) = (2500 – 150) / 40000 = 0.059
Most cost-effective contermeasure

19. HVAC System
Temperature
sensor
MCU
Display
Heat System
Conditioning
System
Input
• Provides heating and cooling
• Supports desirable temperature
• Input: [+5; +40] C0
, int
• Displays current temperature
• Displays desirable temperature

20. Failure Mode and Effect Analysis
Function or
Process Step
Failure
type
Potential
Impact
Sev
erity
Potential
Causes
Occur
rence
Detection
Mode
Detecta
bility
RPN
Temperature
sensor.
Measures
the
environment
temperature
Sensor
failure
Overheating
or constant
cooling
10 Wrong usage
environment,
old sensor,
poor sensor
quality
2 Subjective
temperature
measurement,
thermometer
4 80
Recommend
ed Actions
Responsibili
ty
Target
Date
Action taken Severity Occure
nce
Detecta
bility
RPN
Add extra
sensor as
duplicated
Engineering ASAP If difference between
sensors – shut down
the system
3 2 1 6

21. Quality Function Deployment
How to satisfy
customer wants
Relationships
Matrix
What the
customer wants
Target values
Technical evaluation
Competitive
assessment
Correlation
MatrixCustomer
Importance
ratings
Weighted
rating
• Customer focused
• Allows Competitor Analysis
• Development priorities
oriented on end-user needs
• Formal Validation
• Shows design flaws and
dependencies

22. Correlations:
Strong Positive
Positive
Strong Negative
Negative
+
+
_
_
Relationships:
Strongest= 10
Strong= 7
Fair= 4
Weak= 1
●●
●
•
https://youtu.be/u9bvzE5Qhjk

23. Fault Tree Analysis
• Displays all possible causes that leads to fault
• A fault can be replaced by risk
• If we know all possible causes of the risk – we
can remove or prevent them

24. Car hits an object
AND
Driver doesn’t see
the object
Car fails to break
OR OR
Driver
drunk
Object is
not visible
from car
Driver
asleep
Car
going
too fast
Weak
brakes
https://www.smartdraw.com/fault-tree/examples/fault-tree-example-vehicle-collision/

25. Wanna know more?
• Pragmatic Risk Analysis and Management
– https://rbcs-us.com/site/assets/files/1159/a-case-study-in-risk-based-
testing.pdf
• Product Risk Management:
– http://www.erikvanveenendaal.nl/NL/files/e-book%20PRISMA.pdf
• Fault Tree Analysis / Event Tree Analysis
– https://www.amazon.com/Fault-Analysis-Primer-Clifton-
Ericson/dp/1466446102
– https://www.amazon.com/Risk-Assessment-Tools-Techniques-
Applications/dp/047089203X

26. Q & A