This presentation propose risk base assessment for testing in project. It's driven by defect density and task complexity.

1. MOTOROLA SOLUTIONS
09 Oct 15
Risk-based
Testing & Quality
management in a
programproject.
© 2015 Motorola Solutions, Inc.
ARTUR GORSKI

2. Agenda
1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.

3. Agenda
1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.

4. © 2015 Motorola Solutions, Inc.

5. IDEAL IMAGE
1. We know the number of undiscovered defects in each module of the product.
2. We know distribution of Critical, Not-Critical and Cosmetic defects of
undetected faults.
3. We know tests automation robustness for regression testing.
4. We know the number of defects that won`t be detected without e.g. SIT.
© 2015 Motorola Solutions, Inc.

6. Agenda
1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.

7. Risk #1 - # of undiscovered defects
© 2015 Motorola Solutions, Inc.
Due to the fact that development introduces defects,
there is a possibility of finding them in a late phases,
what can interfere with quality gates and delay release.

8. Let`s use historical data related to our project/product.
This is example data
© 2015 Motorola Solutions, Inc.

9. And pick those that had the same testing process
And where estimates related to implementation tasks
were reliable and available.
This is example data
© 2015 Motorola Solutions, Inc.

10. Gather the data and calculate actual injection rate
(# of defects injected per 1 ideal day of implementation).
Release Dev [ID]
# of sourced
defects
Real Inj.
Rate
Improvement
Project C 500 250 0.5 Baseline
Project E 800 320 0.4 20%
Actual project 0.32 20% (planned)
This is example data
© 2015 Motorola Solutions, Inc.

11. Compare parameters values of a static model, which
will help you to judge whether the conditions of
particular modules are better or worse.
This is example data
© 2015 Motorola Solutions, Inc.

12. Compare parameters values of a static model, which
will help you to judge whether the conditions of
particular modules are better or worse.
This is example data
© 2015 Motorola Solutions, Inc.

13. Now… what are the main factors that influence number
of injected defects ?
Technical complexity
Avg. injection rate
Estimate
Static model
© 2015 Motorola Solutions, Inc.

14. Technical Complexity Assessment
From 1 to 5 (1 – the easiest)
Tester Developer Architect
3 2 4 5 4 4
© 2015 Motorola Solutions, Inc.
This is example data

15. Technical Complexity Assessment
From 1 to 5 (1 – the easiest)
Count average
difference
© 2015 Motorola Solutions, Inc.
This is example data

16. Average Injection Rate is taken from static model
This is example data
© 2015 Motorola Solutions, Inc.

17. Estimates are exported from Version1 and
compared with burn down chart.
© 2015 Motorola Solutions, Inc.
This is example data

18. So we are having technical complexity, avg. inj. Rate and
estimates.
Let`s simulate # of defects that were injected.
This is example data
© 2015 Motorola Solutions, Inc.

19. What is Level Of Trust ?
It`s a level of confidence in the received data.
From 1 to 6 (1 - complete lack of trust)
LoT should be high (5/6)
LoT should be moderate
© 2015 Motorola Solutions, Inc.
This is example data

20. in a case of technical complexity
LoT should be high (5/6)
LoT should be low
Count
average
difference
LoT should be moderate
is moderate
© 2015 Motorola Solutions, Inc.
This is example data

21. in a case of avg. injection rate
think about border values and count disorder
LoT = 5
LoT = 3
[(Max – Avg)/ Avg] * 100%
© 2015 Motorola Solutions, Inc.
This is example data

22. In practice…
The lower the value the more disorders given input
AND
the higher probability of choosing disordered value
during simulation.
LoT Value Disorder in %
6 0%
5 20%
4 40%
3 60%
2 80%
1 100%
© 2015 Motorola Solutions, Inc.

23. Example
Estimate = 10
- 60% disorder + 60% disorder
Given value
- 20% disorder + 20% disorder
LoT_1 = 3
LoT_2 = 5
© 2015 Motorola Solutions, Inc.

24. When the parameters are set let`s run simulation.When the parameters are set, let`s run the simulation.
# of simulations
After each simulation each total
number of defects is stored.
This is example data
© 2015 Motorola Solutions, Inc.

25. When the parameters are set let`s run simulation.
Monte Carlo
Each result is stored
When simulation is finished a histogram of outputs is generated.
The lowest
The highest
The most probable
© 2015 Motorola Solutions, Inc.
This is example data

26. When the parameters are set let`s run simulation.Then cumulative representation is constructed
Monitor and control found vs. injected defects in a module.This is example data
© 2015 Motorola Solutions, Inc.
Injection and found curves for P1F1

27. When the parameters are set let`s run simulation.I ‘know’ how many defects are left, but ‘which’ ones ?
© 2015 Motorola Solutions, Inc.
This is example data
Injection and found curves for P1F1

28. When the parameters are set let`s run simulation.Now we can assess the risk #1
This is example data
© 2015 Motorola Solutions, Inc.
Injection and found curves for P1F1

29. When the parameters are set let`s run simulation.Probability and Impact are used to assess the overall risk
Summary of Risk #1 for each module:
P1/F1
P2/F2
P3/F3
Etc.
© 2015 Motorola Solutions, Inc.

30. When the parameters are set let`s run simulation.What can we do with the information ?
Summary of Risk #1 for each module:
P1/F1
P2/F2
P3/F3
Etc.
 Prioritize execution of test cases
 Deliver information to key
stakeholders about actual product
quality
 Add/Remove testing effort
 etc.
© 2015 Motorola Solutions, Inc.

31. When the parameters are set let`s run simulation.How do we check whether the prediction is correct ?
 Wait for feedback from SIT, FAT, UAT, Client usage etc. 
 Use Software Reliability Engineering e.g. Weibull distributions
CDF: F(t) = 1-e-(t/c)^m
PDF: f(t) = (2/t)(t/c)me -(t/c)^m
Metrics and Models in Software Quality Engineering, 2nd Edition [Chapter 7] Wikipedia.org
© 2015 Motorola Solutions, Inc.

32. When the parameters are set let`s run simulation.How do we check whether the prediction is correct ?
SRE
This is example data
© 2015 Motorola Solutions, Inc.
Injection and found curves for P1F1

33. Agenda
1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.

34. Divide your requirements/specifications
Requirements/Specifications
# of Automatable
# of Non automatable
Module automatability
# of Automated
# of Not Automated
Automation status of a module
# of Non automatable + # of Not Automated Risk impact
Risk probability
Risk level
© 2015 Motorola Solutions, Inc.

35. Example data
Automatability
Automation status
This is example data
P2/F2 Automatability
P2/F2 Automation Status P3/F3 Automation Status
P3/F3 Automatability
© 2015 Motorola Solutions, Inc.

36. Example data
This is example data
P2/F2 Automatability
P2/F2 Automation Status P3/F3 Automation Status
P3/F3 Automatability
© 2015 Motorola Solutions, Inc.

37. Risk overview
Risk #2:
Due to the fact that defects can be injected
into already tested areas during on-going
development or during defects fixing there is
a possibility of delivering a feature with
failing functionalities covered by
requirements what can decrease customer
satisfaction.
Probability:
Automatability level and Automation Status
e.g. 98% and 77%
Impact:
Number of not automated specifications
e.g. 15
© 2015 Motorola Solutions, Inc.
This is example data

38. Use probability and impact to assess the risk
Summary of Risk #2 for each module:
P2/F2
P3/F3
etc.
© 2015 Motorola Solutions, Inc.

39. What can we do with the information ?
Summary of Risk #2 for each module:
P2/F2
P3/F3
etc.
 Change automation strategy to
increase automatability of a module
 Invest more and increase
automation level
 Plan proper manual regression
 etc.
© 2015 Motorola Solutions, Inc.

40. Agenda
1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.

41. Why some of defects are non-detectable during your
testing phase ?
 Some will be injected after your`s testing phase;
 Some require software-hardware integration;
 Some require integration with other systems;
 Some require clients` environment;
 Some require specific configuration;
 Some require specific perspective and higher independency;
 Some require real devices not simulators;
 etc.
© 2015 Motorola Solutions, Inc.

42. S
Assess defects that escaped from previous release
Around 61% were Impossible or Hard to find during our testing phase
This is example data
© 2015 Motorola Solutions, Inc.

43. Risk overview
Risk #3:
Due to the fact that some of the defects are related to hardware, software and hardware integration, real user
environment, systems integration and are impossible to be detected by simulators there is a possibility of
detecting critical defects (that can mask other defects) during e.g. SIT testing phase what can interfere with
quality gates and delay release of a program.
Probability:
Activities and its results done by other testing teams, during demos, dependency to hardware and other system
etc.
Impact:
61% of undiscovered defects
© 2015 Motorola Solutions, Inc.

44. Summary of Risk #3 for each module:
P2/F2
P3/F3
etc.
Use probability and impact to assess the risk
© 2015 Motorola Solutions, Inc.

45. What can we do with the information ?
Summary of Risk #3 for each module:
P2/F2
P3/F3
etc.
 Inform stakeholders about risk
 Plan early system integration
 Built new skills in your team
 etc.
© 2015 Motorola Solutions, Inc.

46. Agenda
1. Problem description.
2. Risk 1 - # of undiscovered defects.
3. Risk 2 - # of failing customer scenarios.
4. Risk 3 - # of non-detectable defects.
5. Summary.
© 2015 Motorola Solutions, Inc.

47. When the parameters are set let`s run simulation.Plan risks reactions and manage the risks.
Feature Risk #1 Risk #2 Risk #3 Risk reactions
P2/F2 accept
P3/F3
1. Invest in exploratory
testing
2. Plan proper manual
regression
3. Plan ESI with SIT
team
P1/F1 …
This is example data
© 2015 Motorola Solutions, Inc.

48. © 2015 Motorola Solutions, Inc.
Any questions ?
ARTUR GORSKI
pl.linkedin.com/in/gorskiartur
artek.gorski@gmail.com