1. Киев 2017Risk based testing in reality: How to make it works on your project?
Risk based testing in reality:
How to make it works on your project?

2. Киев 2017
Speaker Info
• 7 years in IT in roles BA & QA:
Financial Services Industry, Sport Industry
• ISTQB Trainer
Risk based testing in reality: How to make it works on your project?
IRYNA ZHYLINSKA
LEAD SOFTWARE TEST ENGINEER
EPAM

3. Киев 2017
Agenda:
 is Risk-Based testing approach?
 values it brings?
 metrics it gives?
WHAT
HOW  to implement Risk-based testing
on New Project
on existed (on-going Project)
 pros and cons risk based approach launching on your Project
Risk based testing in reality: How to make it works on your project?

4. Киев 2017
WHAT: is Risk-Based testing approach?
Risk based testing in reality: How to make it works on your project?
KEY TERMINOLOGY
POSSIBLE NEGATIVE RISK
+ = Can we test
It?
OUTCOME
PRODUCT
Quality
PROJECT
YES
NO

5. Киев 2017
WHAT: is Risk-Based testing approach?
Risk based testing in reality: How to make it works on your project?
KEY TERMINOLOGY
LIKELYHOOD IMPACT Risk Priority Number
x
=
RBCS link Article: "Risk Based Testing: What It Is and How You Can Benefit" by Rex Black

6. Киев 2017
WHY: Risk Based approach values
… for any system infinite # of tests exists
…so you have to prioritize what is matter to
test
and you have to measure what you test
but you DO NOT have forever to test
Risk based testing in reality: How to make it works on your project?

7. Киев 2017
HOW: implement Risk-Based approach
Risk based testing in reality: How to make it works on your project?
New project On-going project
 Just started
 No history
 Ongoing
 > 2-3 years history
Practical Examples:

8. Киев 2017
HOW: Risk-Based testing process looks like?
 Risk Identification
 Risk Assessment
 Risk Management
 Risk Mitigation
Risk based testing in reality: How to make it works on your project?
OUTPUT
Quality RISKs List
RPN
Test Prioritization
Correct Effort allocation
Defined Scope
STEP
Execute
Report
Analyze
Allocate new Risks
Monitor and Control

9. Киев 2017
Risk-Based implementation: Risk Identification
Risk based testing in reality: How to make it works on your project?
[-] Poor requirements
[-] Lots of changes
[-] No past experience
[-] Busy Stakeholders
[-] No Trust
Goal: Quality Risks List
New project
 Identify + Communicate
 Present
 Apply
 Learn DOMAIN (!)
 Google
 Brainstorming
 Establish standartized KB
 Ask RIGHT questions in RIGHT place
Focus on YOUR Team affected Area
Identify + Establish + Communicated
[+] Business availability
[+] No defined Processes
[+] Time

10. Киев 2017
Risk-Based implementation: Risk Identification
Risk based testing in reality: How to make it works on your project?
Goal: Quality Risks List
Ongoing project
 Find ‘the RIGHT Guy’
 Investigate
 Brainstorm and Identify
bottlenecks
 Identify Risks
Prepare to present
[-] Low Business availability
[-] Documentation: poor/surplus
[-] Custom process
[+] Trust
[+] Domain expertise
[+] Available past experience
 Define Implementation Plan
 Present Solution
Prepare to implement
[?] Existed test cases

11. Киев 2017
Risk-Based implementation: Risk Identification
Risk based testing in reality: How to make it works on your project?
Ongoing projectNew project
Define Quality Risks Catalogue
Maintenance Plan for Risks
(Vision)
Risks Reviews
Define Implementation plan
• Collect data from the past
• Trace past data with identified Risks
• Define missed area
• Allocate time and Resourses
READY TO MOVE TO RISK ASSESMENT
Quality Risks Catalogue

12. Киев 2017
Risk-Based implementation: Risk Identification
Risk based testing in reality: How to make it works on your project?
RESULT = Risks Check List
System response > 20 ms under full load
Patches and releases won’t install correctly
Incorrect exchange rate for currency pair GBP:CNY
User failed to log in and authentication failed
User find interface too complex
Performance
Supportability
Functional
Functional
Usability
US-101
TT-10
US-5
US-65
NA
TC-15
TC-25
TC-35
TC-88
NA
yes
no
yes
yes
NA
User
Story
Test
Case
Is
Aut.
Note: configure your tools to pick all this data automatically
Link for Templates: RBCS: Generic Quality Risks List and General Quality Risk Categories
In case on-going project better to pick
up existed information from the begining

13. Киев 2017
Risk-Based implementation: Risk Assessment
Risk based testing in reality: How to make it works on your project?
[-] Not clear requirements
[-] Not fully defined expectation
[-] No past experience
Goal: assign Risk Priority Number (RPN)
New project
[+] Right people available
 Provide presentation
 Make sure that all understand WHAT Likelihood
and Impact mean
 All understand the Measurement Scale
Explain the RULES
Execute Assessment
 RIGHT people involved
 Appropriate time allocated
 Establish Plan how to deal with ‘open questions’

14. Киев 2017
Risk-Based implementation: Risk Assessment
Risk based testing in reality: How to make it works on your project?
Goal: assign Risk Priority Number (RPN)
Ongoing project
[-] Formed Standards of Priority Scale
[+] Available past experience
 Analyse Past Project Expirience
Execute Assessment
 People involved (!)
 Appropriate time allocated
 Establish Plan how to deal with ‘open
questions’
 Provide presentation
 Make sure that all understand WHAT Likelihood
and Impact mean
 All understand the Measurement Scale
Explain the RULES

15. Киев 2017
Risk-Based implementation: Risk Assessment
Risk based testing in reality: How to make it works on your project?
RESULT = Risk Check List with RNP assessed
1. System response > 20 ms under full load
(performance)
2. Patches and releases won’t install correctly
(supportability)
3. Incorrect exchange rate for currency pair
GBP:CNY (functional)
4. User failed to log in and authentication failed
(functional)
5. User find interface too complex (usability)
3
4
2
4
4
Likelihood Impact Current
RPN
5
1
1
2
5
15
4
2
8
20
Scale
1 =Critical
2 = Major
3 = Medium
4 = Low
5 = Very Low
Past
RNP
12-14
5
2-3
6
0
Analysed expirience -> form RNP based on
existed data

16. Киев 2017
Risk-Based implementation: Risk Mitigation
Risk based testing in reality: How to make it works on your project?
[1-5] Extensive Run ~ 50 % from total test design, implementation end execution effort
a large number of tests that are both broad and deep, exercising combinations and variations of
interesting conditions.
[6-10] Broad Run ~ 25-35 % from total test design implementation end execution effort
a medium number of tests that exercise many different interesting conditions
[11-15] Cursory Run ~ 5-15 % from total test design implementation end execution effort
a small number of tests that sample the most interesting conditions
[16-20] Opportunity ~ 5 % from total test design implementation end execution effort
Leverage other tests or activities to run a test or two of an interesting condition (small investment of time
and effort)
[21-25] Report bugs
MAKE sure that EVERY Stakeholder understand definitions

17. Киев 2017
Risk-Based implementation: Risk Mitigation
Risk based testing in reality: How to make it works on your project?
RESULT = Extend of testing assigned
1. System response > 20 ms under full load
(performance)
2. Patches and releases won’t install correctly
(supportability)
3. Incorrect exchange rate for currency pair
GBP:CNY (functional)
4. User failed to log in and authentication failed
(functional)
5. User find interface too complex (usability)
RPN
15
4
2
8
20
Extend of
testing
C
E
E
B
O
TCs
Suite-15
Suite-18
Suite-1
Suite-2
CR-5
US-101
TT-10
US-5
US-65
NA
US
Issues
I-5, I-9
I-15
I-25
I-65
I-8
TRACING
yes
no
yes
yes
yes
Is
Aut.
Extend of testing
1-5 = [E] = Extensive
6-10 = [B] = Broad
11-15 = [C] = Cursory
15-20 = [O] = Opportunity
21-25 = [RP] = Report Bugs

18. Киев 2017
Risk-Based implementation: Risk Mitigation
Risk based testing in reality: How to make it works on your project?
Goal: allocate test effort
[-] Mixed coverage
[-] Catalogue Maintenance problems
[-] Miss issues (RB, O)
[+] Keep test FOCUS on WHAT is METTER
[+] Limit test afford with RNP
 1 test case covers ONLY 1 entity
 Automate Risk Catalog
 Establish Team Rules
 Allocate time for Experience based testing
New project & Ongoing Project
How to build Report with JIRA: Atlassian Support/JIRA Software Cloud/JIRA Knowledge Base
Atlassian Marketplace: qTest for JIRA Test Management

19. Киев 2017
Risk-Based implementation: Risk Management
Risk based testing in reality: How to make it works on your project?
Goal: analyze, monitor and control
[-] Catalogue Maintenance
[-] Process Maintenance
[+] Time Constraint problem Resolved
(breadth-first and depth-first)
[+] Bug-fix efforts focused on high-risk
problems
[+] Now we known the Scope
 Use Retro
 Build and Analyze Test Report based on
Risks
 Communicate the process
New project & Ongoing Project

20. Киев 2017
Risk-Based implementation: Metrics
Risk based testing in reality: How to makes it works on your project?
Goal: use metrics to improve the process
• % of executed test cases per Risk item
• % of failed/passed/unexecuted test cases per Risk item
New project & Ongoing Project
Test Execution metrics
 % defects identified in 1 Risk item
 % defect identified/fixed per one Risk item in one
iteration
 % defect fount in the 1st – 2nd – 3rd Iterations –
compere
 Defect metrics to establish Acceptance ‘Go’ ‘Not-Go’
Defect metrics

21. Киев 2017
POSSIBLE NEGATIVE RISK
+ =
LIKELYHOOD IMPACT RPN
x =
HOW: Risk-Based testing process looks like?
 Risk
Identification
 Risk
Assessment
 Risk
Mitigation
Extend
of
testing
Extensive Broad Cursory
or or
Opportunity
or =
 Risk
Management
Execute Assign New RisksFix Report
+ + +

22. Киев 2017
WHAT: values Risk Based approach brings?
Risk based testing in reality: How to make it works on your project?
 team knows the focus and domain
 “find the scary stuff first”
 “pick the right tests out of the infinite cloud of possible tests”
 “release when risk of delay balances risk of dissatisfaction”
 “give up tests you worry about the least”
All these BENEFITS allows:
 the test team to operate more efficiently and in a targeted fashion, especially in time-
constrained and/or resource-constrained situations

23. Киев 2017Risk based testing in reality: How to make it works on your project?
ira.zhylinska@gmail.com