Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
05/12/2016 1
Eric RIOU du COSQUER
Minsk, November 24th 2015
Put Risk Based Testing in
place, right now!
05/12/2016 2
You took the risk to attend my presentation
Busines Analyst / Product Owner
Project Manager
Test Manager
Func...
05/12/2016 3
Eric RIOU du COSQUER, erdc@certilogtest.com
• Business Analysis www.iqbba.org
• Member of the executive commi...
05/12/2016 4
The goal is to explain how to implement a Risk Based Testing
approach based on PRISMA® (Product RIsk MAnageme...
05/12/2016 5
Testing, Risk, and Risk Based Testing
Introduction
05/12/2016 6
Main activities (after ISTQB)
What is testing ?
Planning
Control
Closure
Acceptance
System
Integration
Compon...
05/12/2016 7
Definitions (ISTQB)
Risk
• A factor that could result in future negative
consequences; usually expressed as i...
05/12/2016 8
Definition
Risk Based Testing
• An approach to testing to reduce
the level of product risks and
inform stakeh...
05/12/2016 9
A general risk management approach applied to product risks
Risk Management Basics
05/12/2016 10
A process with 4 main activities
Risk
Management
Risk
assessment
Identification
Analysis
Risk control
Mitiga...
05/12/2016 11
The result is a list of risks
• Advice: 30 risks max !
1/4 Risk Identification
Risks Type
Risk 1 Fonctionnal...
05/12/2016 12
Define Likelihood and Impact for each risk, and then a risk
level
• Risk Level = Probability * Impact
2/4 Ri...
05/12/2016 13
The risk level calculation may be supported by a table
2/4 Risk Analysis
05/12/2016 14
Implement actions to reduce the risks
• Four mains options
1. Mitigate the risk through preventive measures ...
05/12/2016 15
Periodically review the risk status , identify new risks and
communicate
4/4 Risk monitoring
Risks Type Prob...
05/12/2016 16
A practical approach, step by step
RBT approach
based on PRISMA®
(Product RISk Management)
05/12/2016 17
The decision to implement an RBT
approach must be made
#1
RBT Selection
05/12/2016 18
Possible insights
Exhaustive testing is impossible
The allocated test design and execution time and
budget i...
05/12/2016 19
The right people to be involved must
be identified
#2
Stakeholders
identification
05/12/2016 20
The Test Manager must select different kind of stakeholders
Who should be involved in the RBT
process ?
On t...
05/12/2016 21
PRISMA provides a checklist for stakeholders identification
Who should be involved in the RBT
process ?
- Pr...
05/12/2016 22
A first list of risks must be created
#3
Risk identification
05/12/2016 23
Different techniques can be combined
How to involve the selected stakeholders in
the risk identification ?
•...
05/12/2016 24
The initial set of product risks must
be improved
#4 Risk triage or
extended
identification
05/12/2016 25
Review the list and check against requirements
• Remove the less relevant risk from the list
• What to do wi...
05/12/2016 26
The impact of each risk needs to be rated
#5
Impact Rating
05/12/2016 27
PRISMA® suggested factors
1. Critical areas (damage, cost and consequences of failure)
2. Visible areas (ext...
05/12/2016 28
The likelihood of each risk needs to be rated
#6
Likelihood Rating
05/12/2016 29
PRISMA® suggested factors
1. Complexity
2. Size
3. Number of changes
4. New technology and methods
5. Inexpe...
05/12/2016 30
Once impact and likelihood are scored,
the risks are included in a Matrix
#7
Risk Matrix creation
05/12/2016 31
Impact and Likelihood are scored for each risk
• Each risk may be rated by different profiles
• Impact: busi...
05/12/2016 32
Each risk will be positioned in a matrix
What is the Product Risk Matrix ?
IIV
II IIII
IIIII
LikelihoodofDef...
05/12/2016 33
IIV
Consider the following advice
1. Avoid the central circle
2. Try not to have all the risks in the same a...
05/12/2016 34
The test approach will be based
on the risk distribution
#8
Test approach and Test
techniques selection
05/12/2016 35
Impact and Likelihood help you focus on the right level(s)
How to allocate the test effort on the
different ...
05/12/2016 36
This question should be adressed for each test level
How to select the right techniques and
define the assoc...
05/12/2016 37
This question should be adressed for each test level
How to select the right techniques and
define the assoc...
05/12/2016 38
The traceability from risks to test cases
is implemented
# 9
Test Design… and
Execution
05/12/2016 39
Use the traceability
How to reach the final Risk Based Test
Execution step ?
Product Risk Requirement Test C...
05/12/2016 40
The risk likelihood and impact
must be reviewed based on
the test execution results
#10
Risk Based reporting...
05/12/2016 41
Update it !
What to do with the Product Risk Matrix
Product Risk Requirement Test Cases
Test
Execution
Resul...
05/12/2016 42
Increase your knowledge in RBT and implement it right now!
What next ?
05/12/2016 43
And at any time!
RBT is everywhere in the test
process
05/12/2016 44
The Best seller about RBT
• ISBN 9789490986070
Sources
05/12/2016 45
With your own Excel file or the PRISMA® tool
The method can be tooled
05/12/2016 46
ISTQB Advanced Level Test Analyst Syllabus
Additional Sources
• http://www.istqb.org/downloa
ds/send/10-adva...
05/12/2016 47
TMMi
• http://www.tmmi.org/wp-
content/uploads/2016/09/T
MMi.Framework.pdf
Additional Sources
05/12/2016 48
Eric RIOU du COSQUER
erdc@certilogtest.com
www.certilogtest.com
Thank you !
Upcoming SlideShare
Loading in …5
×

Put Risk Based Testing in place right now!

658 views

Published on

Доклад Eric Riou du Cosquer на конференции SQA Days-20. 24-26 ноября 2016. Минск
www.sqadays.com

Published in: Education
  • Be the first to comment

Put Risk Based Testing in place right now!

  1. 1. 05/12/2016 1 Eric RIOU du COSQUER Minsk, November 24th 2015 Put Risk Based Testing in place, right now!
  2. 2. 05/12/2016 2 You took the risk to attend my presentation Busines Analyst / Product Owner Project Manager Test Manager Functional or Technical Tester Software Quality and Testing consultant Sales person About you
  3. 3. 05/12/2016 3 Eric RIOU du COSQUER, erdc@certilogtest.com • Business Analysis www.iqbba.org • Member of the executive committee • Requirements Engineering www.reqb.org • Member of the executive committee • International Software Testing www.istqb.org • General Secretary from 2011 to 2015, France Representative afterwards • French Software Testing Qualification Board www.cftl.fr • Manager since 2013 • Test organizations assessment www.tmmi.org • Lead Assessor since 2015 About me
  4. 4. 05/12/2016 4 The goal is to explain how to implement a Risk Based Testing approach based on PRISMA® (Product RIsk MAnagement) Introduction Risk Management Basics RBT approach What next? Summary Agenda
  5. 5. 05/12/2016 5 Testing, Risk, and Risk Based Testing Introduction
  6. 6. 05/12/2016 6 Main activities (after ISTQB) What is testing ? Planning Control Closure Acceptance System Integration Component 1 Analysis and Design Implementation and Execution Evaluation & Reporting Planification Closure Control
  7. 7. 05/12/2016 7 Definitions (ISTQB) Risk • A factor that could result in future negative consequences; usually expressed as impact and likelihood Product Risk • A risk directly related to the test object Project Risk • A risk related to management and control of the (test) project, e.g. lack of staffing, strict deadlines, changing requirements… What is a risk ?
  8. 8. 05/12/2016 8 Definition Risk Based Testing • An approach to testing to reduce the level of product risks and inform stakeholders of their status (…). It involves the identification of product risks and the use of risk levels to guide the process What is « RBT » ? (Risk Based Testing)
  9. 9. 05/12/2016 9 A general risk management approach applied to product risks Risk Management Basics
  10. 10. 05/12/2016 10 A process with 4 main activities Risk Management Risk assessment Identification Analysis Risk control Mitigation Monitoring What does the general risk management approach consist in ?
  11. 11. 05/12/2016 11 The result is a list of risks • Advice: 30 risks max ! 1/4 Risk Identification Risks Type Risk 1 Fonctionnal Risk 2 Security Risk 3 Fonctionnal Risk 4: Reliability …
  12. 12. 05/12/2016 12 Define Likelihood and Impact for each risk, and then a risk level • Risk Level = Probability * Impact 2/4 Risk Analysis Risks Type Likelihood Impact Level Risk 1 Fonctionnal Risk 2 Security Risk 3 Fonctionnal Risk 4: Reliability … … … … …
  13. 13. 05/12/2016 13 The risk level calculation may be supported by a table 2/4 Risk Analysis
  14. 14. 05/12/2016 14 Implement actions to reduce the risks • Four mains options 1. Mitigate the risk through preventive measures to reduce likelihood and/or impact 2. Make contingency plans to reduce impact if the risk becomes an actuality 3. Transfer the risk to some other party to handle 4. Ignore and accept the risk, which means doing nothing but wait and see whether the problem occurs or not. • Mitigation with testing • Associate test cases to the risks 3/4 Risk Mitigation
  15. 15. 05/12/2016 15 Periodically review the risk status , identify new risks and communicate 4/4 Risk monitoring Risks Type Proba. Impact Action Status Level Risk 1 Fonctionnal Risk 2 Security Risk 3 Fonctionnal Risk 4: Reliability … … … … … New Risk
  16. 16. 05/12/2016 16 A practical approach, step by step RBT approach based on PRISMA® (Product RISk Management)
  17. 17. 05/12/2016 17 The decision to implement an RBT approach must be made #1 RBT Selection
  18. 18. 05/12/2016 18 Possible insights Exhaustive testing is impossible The allocated test design and execution time and budget is always reduced The specifications and requirements may not cover the overall set of expected caracteristics The quality and success of a product depend on the final users and customers view How to (be) convice(d) to implement an RBT approach ?
  19. 19. 05/12/2016 19 The right people to be involved must be identified #2 Stakeholders identification
  20. 20. 05/12/2016 20 The Test Manager must select different kind of stakeholders Who should be involved in the RBT process ? On the vendor side On the customer side External stakeholder • End user (client of the customer) • Other organizations (regulatory entities,…) • Customer representatives (called “Business”) • Project sponsors • End users (from the customer company) • Installation and Operations personnel • Testers and Quality Assurance staff • Project managers • Business and System Analysts • Developers and architects • DBA • GUI designers • Technical writers • Testers and Quality Assurance staff
  21. 21. 05/12/2016 21 PRISMA provides a checklist for stakeholders identification Who should be involved in the RBT process ? - Project manager - Business experts - Designers - Testers - Client / sponsor - End users - Usability experts - Operations - Maintenance team - Security - Safety services - Inspectors - Support / helpdesk - Manufacturing - Marketing - Legal - Professional bodies - Special interest groups - Technology experts - Marketing - Customers - System development - Quality assurance - Regulatory bodies
  22. 22. 05/12/2016 22 A first list of risks must be created #3 Risk identification
  23. 23. 05/12/2016 23 Different techniques can be combined How to involve the selected stakeholders in the risk identification ? • Requirements based • Interviews • Workshops and Brainstorming sessions Risks Type Risk 1 Fonctionnal Risk 2 Security Risk 3 Fonctionnal Risk 4: Reliability … Same result as above
  24. 24. 05/12/2016 24 The initial set of product risks must be improved #4 Risk triage or extended identification
  25. 25. 05/12/2016 25 Review the list and check against requirements • Remove the less relevant risk from the list • What to do with • A risk but no requirement • A requirement but no risk How to keep the most relevant risks in the list ? Product Risk Requirement ID Product Risk Risk Type Requirement 01 Customer cannot start the transaction at another bank Functionality Customer shall be able to perform a transaction at another bank 02 Customer not issued with receipt at the end of the transaction Functionality Customer shall receive a receipt at the end of the transaction 03 The system is unavailable to the customer for longer than two hours Reliability System shall be available to customers 24/7 … … Example of a set of product risks for an after Pinkster]
  26. 26. 05/12/2016 26 The impact of each risk needs to be rated #5 Impact Rating
  27. 27. 05/12/2016 27 PRISMA® suggested factors 1. Critical areas (damage, cost and consequences of failure) 2. Visible areas (external visibility of a failure) 3. Most used areas 4. Business importance 5. Cost of rework Which factors shall we consider to rate the impact ? Impact Factor Criticity Visibility … Weight 2 1 … Risk 1 5 3 … Risk 2 3 5 … Risk 3 3 2 … … … … …
  28. 28. 05/12/2016 28 The likelihood of each risk needs to be rated #6 Likelihood Rating
  29. 29. 05/12/2016 29 PRISMA® suggested factors 1. Complexity 2. Size 3. Number of changes 4. New technology and methods 5. Inexperience 6. New development vs. re-use 7. Interfacing 8. … Which factors shall we consider to rate the likelihood ? Impact LIkelihood Factor Criticity Visibility … Complexity Size … Weight 2 1 … 1 2 … Risk 1 5 3 … 3 5 … Risk 2 3 5 … 4 1 … Risk 3 3 2 … 2 4 … … … … … … … …
  30. 30. 05/12/2016 30 Once impact and likelihood are scored, the risks are included in a Matrix #7 Risk Matrix creation
  31. 31. 05/12/2016 31 Impact and Likelihood are scored for each risk • Each risk may be rated by different profiles • Impact: business skills • Likelihood: technical skills How to visualize the risk distribution ? Impact Probabilité Factor Criticity Visibility VALUE Complexity Size VALUE Weight 2 1 na 1 2 na Risk 1 5 3 13 3 5 13 Risk 2 3 5 11 4 1 6 … … … … … … …
  32. 32. 05/12/2016 32 Each risk will be positioned in a matrix What is the Product Risk Matrix ? IIV II IIII IIIII LikelihoodofDefects (TechnicalRisks) Impact of Defects (Business Risks) 3 3 15 15 R1 R2 R3 R4 R5
  33. 33. 05/12/2016 33 IIV Consider the following advice 1. Avoid the central circle 2. Try not to have all the risks in the same areas 3. Add a fifth area for safety-critical applications How to ensure a right distribution of the risks ? IIV II IIII IIIII Likelihood Impact3 3 15 15 R1 R2 R4 R5 R5 R7 R6
  34. 34. 05/12/2016 34 The test approach will be based on the risk distribution #8 Test approach and Test techniques selection
  35. 35. 05/12/2016 35 Impact and Likelihood help you focus on the right level(s) How to allocate the test effort on the different levels ? IIV II IIII IIIII Likelihood Impact3 3 15 15 component and Integration level test (focus on technical risk) system and acceptance level test (focus on business risk)
  36. 36. 05/12/2016 36 This question should be adressed for each test level How to select the right techniques and define the associated coverage goals ? IIV II IIII IIIII Likelihood Impact3 3 15 15 Example for the component level Decision coverage (90%) Code inspection Instruction coverage (90%) Instruction coverage (70%)
  37. 37. 05/12/2016 37 This question should be adressed for each test level How to select the right techniques and define the associated coverage goals ? IIV II IIII IIIII Probabilité Impact3 3 15 15 Use Case (incl alternative paths) Decision table Use Case (main path) Equivalence partitioning Use Case (incl alternative paths) Equivalence partitioning Use Case (main path Exploratory testing Example for the acceptance level
  38. 38. 05/12/2016 38 The traceability from risks to test cases is implemented # 9 Test Design… and Execution
  39. 39. 05/12/2016 39 Use the traceability How to reach the final Risk Based Test Execution step ? Product Risk Requirement Test Cases Test Execution Results Defects
  40. 40. 05/12/2016 40 The risk likelihood and impact must be reviewed based on the test execution results #10 Risk Based reporting and Defect correction
  41. 41. 05/12/2016 41 Update it ! What to do with the Product Risk Matrix Product Risk Requirement Test Cases Test Execution Results Defects Defects  Likelihood is increased Passed test cases  Likelihood is decreased New risks ?
  42. 42. 05/12/2016 42 Increase your knowledge in RBT and implement it right now! What next ?
  43. 43. 05/12/2016 43 And at any time! RBT is everywhere in the test process
  44. 44. 05/12/2016 44 The Best seller about RBT • ISBN 9789490986070 Sources
  45. 45. 05/12/2016 45 With your own Excel file or the PRISMA® tool The method can be tooled
  46. 46. 05/12/2016 46 ISTQB Advanced Level Test Analyst Syllabus Additional Sources • http://www.istqb.org/downloa ds/send/10-advanced-level- syllabus-2012/53-advanced- level-syllabus-2012-test- analyst.html
  47. 47. 05/12/2016 47 TMMi • http://www.tmmi.org/wp- content/uploads/2016/09/T MMi.Framework.pdf Additional Sources
  48. 48. 05/12/2016 48 Eric RIOU du COSQUER erdc@certilogtest.com www.certilogtest.com Thank you !

×