SlideShare a Scribd company logo

Risk Based Testing in Agile context - Kees Blokland

Kees did this talk for a client in 2016. It is about the basic principles behind risk based testing - also in Agile context.

1 of 47
Download to read offline
1© 2016
Risk based testing in Agile? context
Huawei, Shenzhen 2016
Kees Blokland
kees.blokland@polteq.com
3© 2016
Question to you:
• What does risk based testing mean for you?
4© 2016
Testers must make choices
What to test and what not to test
What to test thoroughly, what to test superficially
What to test first, what to test later
What to test once, what to test each software build
What to test by senior testers, what by junior testers
What to test scripted, what to test exploratory,
… just to name a few
We cannot test all
possible ways
through the
software
5© 2016
Testers must make choices
What to test and what not to test
What to test thoroughly, what to test superficially
What to test first, what to test later
What to test once, what to test each software build
What to test by senior testers, what by junior testers
What to test scripted, what to test exploratory,
… just to name a few
We base these
choices on risk
6© 2016
What is risk?
7© 2016
Risk defined
WIKIPEDIA
The probability or threat of
quantifiable damage, injury,
liability, loss, or any other
negative occurrence that
is caused by external or
internal vulnerabilities, and
that may be avoided through
pre-emptive action.
8© 2016
Risk defined
Michael Bolton
Some person will suffer
loss, harm, annoyance,
or reduced value
because of a problem in
the product that is
triggered by some
threat.
9© 2016
Risk defined
WIKIPEDIA
The probability or threat of
quantifiable damage, injury,
liability, loss, or any other
negative occurrence that
is caused by external or
internal vulnerabilities, and
that may be avoided through
pre-emptive action.
a system failure
which is bad to a certain
extend
we don’t know for sure it
will happen
we may want to prevent it
10© 2016
BMW needs to call back 10.000 cars in
China due to a software defect
source: computable.nl
Doors open
spontaneously
Gear switches from parked
state and car starts rolling by
itself from the hill
11© 2016
Huge traffic jam due to system failure
source: computable.nl
Huge traffic jams
due to failing
information
boards
12© 2016
Information
still usable…
13© 2016
Testing: Test’s law on software failures
R = L x IRisk Likelihood Impact
NO RISK NO TEST
14© 2016
Risk assessment
WIKIPEDIA
The probability or threat of
quantifiable damage, injury,
liability, loss, or any other
negative occurrence that
is caused by external or
internal vulnerabilities, and
that may be avoided through
pre-emptive action.
What failures of the
product could we
anticipate.
What impact would
these have?
15© 2016
Risk assessment
WIKIPEDIA
The probability or threat of
quantifiable damage, injury,
liability, loss, or any other
negative occurrence that
is caused by external or
internal vulnerabilities, and
that may be avoided through
pre-emptive action.
What would cause faults
in the product that make it
fail like that?
What is the likelihood of
this to happen?
16© 2016
Risk reduction
How can testing help to
reduce the risks
by finding these
(types of) faults so
preventing the failures to
happen in the product?
17© 2016
break down break down
recognizable by
customer / user
recognizable by
development
prioritized on
business value
acceptance criteria
user scenario’s
Feature refinement, break down
18© 2016
break down break down
recognizable by
customer / user
recognizable by
development
prioritized on
business value
acceptance criteria
user scenario’s
Question
• Discuss where to find these in your project
19© 2016
break down break down
What failures could we anticipate.
What impact would these have?
What would cause faults in the product?
What is the likelihood of this to happen?
How can testing help to reduce the risks
by finding faults causing the failures?
20© 2016
break down break down
What failures could we anticipate.
What impact would these have?
Risk assessment
Who represents
the user / client
/ business?
Risk Story Elements
• Some PERSON(S)
– a person, a group, the business, a bystander
• will EXPERIENCE
– bad feelings and a trigger for them
• a PROBLEM
– annoyance, loss, harm, diminished value…
• with respect to SOMETHING DESIRABLE
– capability, reliability, performance…
• that CAN BE DETECTED
– by an oracle (a means of recognizing a problem)…
• in SOME SET OF CONDITIONS
– perhaps invariably, perhaps intermittently…
• because of a VULNERABILITY
• a bug, a missing feature, an inconsistency…
• in the SYSTEM.
– some result, process, component, environment…
© James Bach and Michael Bolton, 2016
© James Bach and Michael Bolton, 2016
Some General Systems Risk Heuristics
• A condition in the world that interacts with your
product may happen rarely or frequently.
• A problem may be detectable or undetectable at the
time it first occurs or begins to occur (relative to our
means of detection).
• A problem may occur in a result, process, component,
or environment.
• In the realm of technology and society, problems fall
into diverse categories: usability, performance,
concurrency, legality, correctness, compatibility, etc.
• A problem may impact a user, a bystander, or the
business that produced the product.
• A problem may cause inconvenience, annoyance, or
harm.
• …
Examples
• sometimes (in a given situation) one of the components OF THE PRODUCT visibly FAILS WITH
RESPECT TO compatibility SUCH THAT bystander IS harmed
• sometimes (in a given situation) some process OF THE PRODUCT visibly FAILS WITH RESPECT
TO legality SUCH THAT business IS harmed
• sometimes (in a given situation) some process OF THE PRODUCT visibly FAILS WITH RESPECT
TO usability SUCH THAT user IS annoyed
• always some result OF THE PRODUCT invisibly FAILS WITH RESPECT TO concurrency SUCH
THAT bystander IS annoyed
• always some process OF THE PRODUCT invisibly FAILS WITH RESPECT TO legality SUCH THAT
bystander IS inconvenienced
• always some result OF THE PRODUCT visibly FAILS WITH RESPECT TO performance SUCH
THAT business IS annoyed
• always some result OF THE PRODUCT visibly FAILS WITH RESPECT TO compatibility SUCH
THAT bystander IS harmed
• sometimes (in a given situation) some result OF THE PRODUCT invisibly FAILS WITH RESPECT
TO correctness SUCH THAT business IS inconvenienced
• always one of the components OF THE PRODUCT invisibly FAILS WITH RESPECT TO
performance SUCH THAT user IS inconvenienced
• always one of the components OF THE PRODUCT visibly FAILS WITH RESPECT TO correctness
SUCH THAT user IS annoyed © James Bach and Michael Bolton, 2016
24© 2016
Question to you: what are your examples?
and: who should we talk with?
25© 2016
Risk assessment
– Discuss the impact
Losing income
Losing customers
Societal
Damage
Claims by users
Repair costs
Damage to company image
Compensation effort and cost
Development and test cost
Clients / Users /
Business perspective
26© 2016
Risk assessment
User story Risk Quality Impact
User story 1
Template
Unable to install client SW, loose customer functional H
Customer does not understand how to operate
(and terminates session)
usability H
User story 2
Upload &
Preview
Preview does not function (customer
disappointed)
functional M
Unable to create a mailing, less customer value performance H
Upload over 3G is taking too long (customer
complains or stops)
functional M
User story 4 Date format incorrect functional L
User story 5 Upload does not start functional H
…
27© 2016
break down break down
What would cause faults in the product?
What is the likelihood of this to happen?
Risk assessment
28© 2016
Risk Analysis Heuristics
• Project factors
– Things going on in projects, among people, may lead to
bugs.
• Technology factors
– The structure and dynamics of technology itself may give
rise to bugs.
• Requirements factors
– Aspects of requirements may indicate or promote the
presence of bugs.
• Operational factors
– The circumstances and patterns of use affect the
probability and impact of bugs.
Risk Analysis Heuristics (for Digital Products)
By James Bach and Michael Bolton v. 2.1
29© 2016
Example of technology factors
logic
platform
Service
boards
Main
NE
linux
lamps
cabling
subrack
…
Service
Board-1
Main b power
Service
Board-n
Service
Board-2
…
Feature map
Customer application scenario
100G extra long haul 1550nm
Protection switching
Automatic laser shutdown
Alarming
AU4-4c channel provisioning
Adding / removing wavelengths
….
Software map
Hardware mapNetwork map
Service board
Asic1
pwr
Otherstuff
Asic 2
Firmware
optics
NE NMS
Here are the
changes made
30© 2016
Risk assessment
– another list for inspiration
Frequency of use
Inexperienced developers
New tools
High time pressure
Complex functions
Many changes
Many interfaces
Error prone functions
31© 2016
What are your examples?
32© 2016
Risk assessment
User story 1:
Template
Q Risk Fault
proba
bility
Unable to
install client
SW, loose
customer
F Install on windows; stable, predictable, hardly
bugs seen before
L
F Install on macos; NEW, no experience, high
time pressure
H
P Image becomes bigger, may cause
performance problems
M
33© 2016
Risk assessment
User story 1:
Template
Q Risk Impact Fault
prob.
Risk
class
Unable to
install client
SW, loose
customer
F Install on windows; stable,
predictable, hardly bugs
seen before
H L M
F Install on macos; NEW, no
experience, high time
pressure
H H H
P Image becomes bigger,
may cause performance
problems
H M M
Impact
H M L
Fault pr.
H H M L
M M M L
L M L L
Risk class
lookup
These are most important!
34© 2016
Do you remember this slide?
35© 2016
Question to you:
• How to test more thorough, more deeply?
36© 2016
How to map risks on tests?
Techniques
Superficial
Implicit
Thorough
Specific
Formal
Multiple
Model based
Medium risk
High risk
Low risk
Test levels
Component Test
+ System Test
+ Acceptance Test
+ System Integr. T.
Special test
All
Performance
Security
Usability
Testers
Junior
Medior
Senior
Exploratory testing
Free format
Session based
Session based by experts
37© 2016
How to map risks on tests? - continued
Tested once
Automated regr. test
Test rounds Multiple
Each
Manual regression test
Review
-
Informal
Formal
Specific
Focus
Code review
-
Pairing
Pairing with testers
Testers
Junior
Medior
Senior
Experts
38© 2016
Question to you:
• What techniques do you know to do more deep
testing?
39© 2016
Do you remember this slide?
40© 2016
What to test first and what later?
In the end execute tests on
low risk components /
features / aspects
First execute tests on
high risk components /
features / aspects
Time
Low severity
defects
undiscovered
Most serious
defects
Low severity
defects
Then execute tests on
medium risk components /
features / aspects
Less serious
defects
Project
deadline
41© 2016
Determining testing strategy
Backlog item list
- Changes
- Defects
- Technical debt
Items considered for
this iteration
Iteration, e.g. 3 weeks
Iteration planning
meeting and/or
clarification /
elaboration meeting
Time 
SE
DE DV
TE
CR
Example
42© 2016
Determining testing strategy
Backlog item list
- Changes
- Defects
- Technical debt
Items considered for
this iteration
Iteration, e.g. 3 weeks
Iteration planning
meeting and/or
clarification /
elaboration meeting
Time 
SE
DE DV
TE
CR
Example
• What ‘tools’ do we have?
– Pair testing
– Code review
– Automated regression testing (/checking)
– Manual regression testing
– Unit testing
– TDD
– Exploratory sessions
– Scripted testing (prepare for automated regression
checks)
– …
43© 2016
Reporting about testing (risk related)
• What information have we gained about the quality of
the product?
– in general and specifically:
– regarding the potential product failures identified in the
risk assessment
• What testing has been done so far?
• What information do we have to determine further
steps?
– how testability of the product influenced testing
– what further testing could be done (related to the risks)
– the cost of further testing (effort and lead time)
Any level:
TE level, TPL level, TM level, …
44© 2016
Reporting about testing (risk related)
• What information have we gained about the quality of
the product?
– in general and specifically:
– regarding the potential product failures identified in the
risk assessment
• What testing has been done so far?
• What information do we have to determine further
steps?
– how testability of the product influenced testing
– what further testing could be done (related to the risks)
– the cost of further testing (effort and lead time)
Example 1:
Software delivery delayed due to complexity problems
SO: not all planned testing could be done
AND: need EVEN MORE TESTING
because delayed delivery increases product risk
Example 2:
Found many bugs so far, which are solved and retested
BUT: more testing is likely to uncover more bugs
SO: need MORE TESTING
because many bugs found is is an indicator of high risk
45© 2016
Considerations on risk based testing
• Risk based testing requires solid testing skills
• Risk based testing requires dialogue with the
client/user/customer
• Risk based testing requires dialogue with developers
• Information from testing can lower the risk level
• Information from testing can ALSO indicate a HIGHER
risk level
46© 2016
Last question to you:
• Do YOU have questions?
47© 2016
谢谢!
Thank you!
kees.blokland@polteq.com

Recommended

Mindful Metrics (QAotHW 2018)
Mindful Metrics (QAotHW 2018)Mindful Metrics (QAotHW 2018)
Mindful Metrics (QAotHW 2018)Dmitry Sharkov
 
Intro to Software Engineering - Software Quality Assurance
Intro to Software Engineering - Software Quality AssuranceIntro to Software Engineering - Software Quality Assurance
Intro to Software Engineering - Software Quality AssuranceRadu_Negulescu
 
Error-Free® Human Performance
Error-Free® Human PerformanceError-Free® Human Performance
Error-Free® Human PerformanceChong Chiu
 
Error-Free® Operation and Human Performance
Error-Free® Operation and Human PerformanceError-Free® Operation and Human Performance
Error-Free® Operation and Human PerformanceChong Chiu
 
CoverMyQuality: Implementing a Quality Program by Rick Neighbarger and Susan ...
CoverMyQuality: Implementing a Quality Program by Rick Neighbarger and Susan ...CoverMyQuality: Implementing a Quality Program by Rick Neighbarger and Susan ...
CoverMyQuality: Implementing a Quality Program by Rick Neighbarger and Susan ...QA or the Highway
 
Root Cause Analysis - methods and best practice
Root Cause Analysis - methods and best practiceRoot Cause Analysis - methods and best practice
Root Cause Analysis - methods and best practiceMedgate Inc.
 
Error-Free® Six Sigma
Error-Free® Six SigmaError-Free® Six Sigma
Error-Free® Six SigmaChong Chiu
 

More Related Content

What's hot

Poka Yoke - Fundamentals by Praneet Surti
Poka Yoke - Fundamentals by Praneet SurtiPoka Yoke - Fundamentals by Praneet Surti
Poka Yoke - Fundamentals by Praneet SurtiPraneet Surti
 
Live Conversation: Cut your customer interview costs by up to 90%
Live Conversation: Cut your customer interview costs by up to 90%Live Conversation: Cut your customer interview costs by up to 90%
Live Conversation: Cut your customer interview costs by up to 90%UserTesting
 
Building a Control Plan
Building a Control PlanBuilding a Control Plan
Building a Control PlanMatt Hansen
 
Corrective & Preventive Action
Corrective & Preventive Action Corrective & Preventive Action
Corrective & Preventive Action Praneet Surti
 
Root Cause Analysis Training for Healthcare Professionals : Tonex Training
Root Cause Analysis Training for Healthcare Professionals : Tonex TrainingRoot Cause Analysis Training for Healthcare Professionals : Tonex Training
Root Cause Analysis Training for Healthcare Professionals : Tonex TrainingBryan Len
 
Phil Koopman's ISSRE 2016 Keynote
Phil Koopman's ISSRE 2016 KeynotePhil Koopman's ISSRE 2016 Keynote
Phil Koopman's ISSRE 2016 Keynoteedgecaseresearch
 
The Seven Deadly Sins of Software Testing
The Seven Deadly Sins of Software TestingThe Seven Deadly Sins of Software Testing
The Seven Deadly Sins of Software TestingTechWell
 
Effective CAPA Implementation in a Management System - Praneet Surti
Effective CAPA Implementation in a Management System - Praneet SurtiEffective CAPA Implementation in a Management System - Praneet Surti
Effective CAPA Implementation in a Management System - Praneet SurtiPraneet Surti
 
Hazop leaders manual final
Hazop leaders manual finalHazop leaders manual final
Hazop leaders manual finalumar farooq
 
IRJET- Faces of Testing Strategies: Why &When?
IRJET- Faces of Testing Strategies: Why &When?IRJET- Faces of Testing Strategies: Why &When?
IRJET- Faces of Testing Strategies: Why &When?IRJET Journal
 
Webinar: Medical Device Security: An Industry Under Attack and Unprepared to ...
Webinar: Medical Device Security: An Industry Under Attack and Unprepared to ...Webinar: Medical Device Security: An Industry Under Attack and Unprepared to ...
Webinar: Medical Device Security: An Industry Under Attack and Unprepared to ...Synopsys Software Integrity Group
 
Control Phase Roadmap (Level 3)
Control Phase Roadmap (Level 3)Control Phase Roadmap (Level 3)
Control Phase Roadmap (Level 3)Matt Hansen
 
When Agile is a Quality Game Changer Webinar - Michael Mah, Philip Lew
When Agile is a Quality Game Changer Webinar - Michael Mah, Philip LewWhen Agile is a Quality Game Changer Webinar - Michael Mah, Philip Lew
When Agile is a Quality Game Changer Webinar - Michael Mah, Philip LewXBOSoft
 
Piloting Solutions: Build the Pilot Plan
Piloting Solutions: Build the Pilot PlanPiloting Solutions: Build the Pilot Plan
Piloting Solutions: Build the Pilot PlanMatt Hansen
 
Closing a Project
Closing a ProjectClosing a Project
Closing a ProjectMatt Hansen
 
Ken Johnston - Big Bugs That Got Away - EuroSTAR 2010
Ken Johnston - Big Bugs That Got Away -  EuroSTAR 2010Ken Johnston - Big Bugs That Got Away -  EuroSTAR 2010
Ken Johnston - Big Bugs That Got Away - EuroSTAR 2010TEST Huddle
 
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010TEST Huddle
 

What's hot (20)

Poka Yoke - Fundamentals by Praneet Surti
Poka Yoke - Fundamentals by Praneet SurtiPoka Yoke - Fundamentals by Praneet Surti
Poka Yoke - Fundamentals by Praneet Surti
 
Live Conversation: Cut your customer interview costs by up to 90%
Live Conversation: Cut your customer interview costs by up to 90%Live Conversation: Cut your customer interview costs by up to 90%
Live Conversation: Cut your customer interview costs by up to 90%
 
It’s a world of bugs after all
It’s a world of bugs after allIt’s a world of bugs after all
It’s a world of bugs after all
 
Building a Control Plan
Building a Control PlanBuilding a Control Plan
Building a Control Plan
 
Corrective & Preventive Action
Corrective & Preventive Action Corrective & Preventive Action
Corrective & Preventive Action
 
Root Cause Analysis Training for Healthcare Professionals : Tonex Training
Root Cause Analysis Training for Healthcare Professionals : Tonex TrainingRoot Cause Analysis Training for Healthcare Professionals : Tonex Training
Root Cause Analysis Training for Healthcare Professionals : Tonex Training
 
Phil Koopman's ISSRE 2016 Keynote
Phil Koopman's ISSRE 2016 KeynotePhil Koopman's ISSRE 2016 Keynote
Phil Koopman's ISSRE 2016 Keynote
 
Remote Usability Testing
Remote Usability TestingRemote Usability Testing
Remote Usability Testing
 
The Seven Deadly Sins of Software Testing
The Seven Deadly Sins of Software TestingThe Seven Deadly Sins of Software Testing
The Seven Deadly Sins of Software Testing
 
Effective CAPA Implementation in a Management System - Praneet Surti
Effective CAPA Implementation in a Management System - Praneet SurtiEffective CAPA Implementation in a Management System - Praneet Surti
Effective CAPA Implementation in a Management System - Praneet Surti
 
Hazop leaders manual final
Hazop leaders manual finalHazop leaders manual final
Hazop leaders manual final
 
IRJET- Faces of Testing Strategies: Why &When?
IRJET- Faces of Testing Strategies: Why &When?IRJET- Faces of Testing Strategies: Why &When?
IRJET- Faces of Testing Strategies: Why &When?
 
Webinar: Medical Device Security: An Industry Under Attack and Unprepared to ...
Webinar: Medical Device Security: An Industry Under Attack and Unprepared to ...Webinar: Medical Device Security: An Industry Under Attack and Unprepared to ...
Webinar: Medical Device Security: An Industry Under Attack and Unprepared to ...
 
Control Phase Roadmap (Level 3)
Control Phase Roadmap (Level 3)Control Phase Roadmap (Level 3)
Control Phase Roadmap (Level 3)
 
When Agile is a Quality Game Changer Webinar - Michael Mah, Philip Lew
When Agile is a Quality Game Changer Webinar - Michael Mah, Philip LewWhen Agile is a Quality Game Changer Webinar - Michael Mah, Philip Lew
When Agile is a Quality Game Changer Webinar - Michael Mah, Philip Lew
 
Piloting Solutions: Build the Pilot Plan
Piloting Solutions: Build the Pilot PlanPiloting Solutions: Build the Pilot Plan
Piloting Solutions: Build the Pilot Plan
 
Closing a Project
Closing a ProjectClosing a Project
Closing a Project
 
Ken Johnston - Big Bugs That Got Away - EuroSTAR 2010
Ken Johnston - Big Bugs That Got Away -  EuroSTAR 2010Ken Johnston - Big Bugs That Got Away -  EuroSTAR 2010
Ken Johnston - Big Bugs That Got Away - EuroSTAR 2010
 
Bug Advocacy
Bug AdvocacyBug Advocacy
Bug Advocacy
 
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010
Doron Reuveni - The Mobile App Quality Challenge - EuroSTAR 2010
 

Similar to Risk Based Testing in Agile context - Kees Blokland

Risk Based Thinking ISO 9001 Presentation.pdf
Risk Based Thinking ISO 9001 Presentation.pdfRisk Based Thinking ISO 9001 Presentation.pdf
Risk Based Thinking ISO 9001 Presentation.pdfHimanshuMishra203021
 
EXTENT-2016: The Future of Software Testing
EXTENT-2016:	 The Future of Software TestingEXTENT-2016:	 The Future of Software Testing
EXTENT-2016: The Future of Software TestingIosif Itkin
 
Implementing Risk Based Thinking in HLS OF ISO 9001:2015 - Praneet Surti
Implementing Risk Based Thinking in HLS OF ISO 9001:2015 - Praneet SurtiImplementing Risk Based Thinking in HLS OF ISO 9001:2015 - Praneet Surti
Implementing Risk Based Thinking in HLS OF ISO 9001:2015 - Praneet SurtiPraneet Surti
 
Industrial Training in Software Testing
Industrial Training in Software TestingIndustrial Training in Software Testing
Industrial Training in Software TestingArcadian Learning
 
Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Inn...
Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Inn...Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Inn...
Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Inn...TEST Huddle
 
Fundamentals of testing
Fundamentals of testingFundamentals of testing
Fundamentals of testingYusran5
 
Risk Driven Testing
Risk Driven TestingRisk Driven Testing
Risk Driven TestingJorge Boria
 
Risk Management
Risk Management Risk Management
Risk Management Kapil Rode
 
UX Lead Product Management
UX Lead Product ManagementUX Lead Product Management
UX Lead Product ManagementJohn Wyatt
 
Fundamentals of testing (1)
Fundamentals of testing (1)Fundamentals of testing (1)
Fundamentals of testing (1)Aziz Chikhly
 
Fundamentals of crowdsourced testing
Fundamentals of crowdsourced testingFundamentals of crowdsourced testing
Fundamentals of crowdsourced testingNicholas Roberts
 
Chapter 1 - Why Testing.ppt
Chapter 1 - Why Testing.pptChapter 1 - Why Testing.ppt
Chapter 1 - Why Testing.pptAsgar Ali Sayed
 
Presentasi fundamental of testing
Presentasi fundamental of testingPresentasi fundamental of testing
Presentasi fundamental of testingEgi Ilham Elnusa
 
18 Jul 2018 - FMEA and Risk Management in Practice
18 Jul 2018 - FMEA and Risk Management in Practice 18 Jul 2018 - FMEA and Risk Management in Practice
18 Jul 2018 - FMEA and Risk Management in Practice Intland Software GmbH
 
Better Software East 2016: Evolving Automated to Continuous
Better Software East 2016: Evolving Automated to ContinuousBetter Software East 2016: Evolving Automated to Continuous
Better Software East 2016: Evolving Automated to ContinuousParasoft
 
Navigating Your Product's Growth with Embedded Analytics
Navigating Your Product's Growth with Embedded Analytics Navigating Your Product's Growth with Embedded Analytics
Navigating Your Product's Growth with Embedded Analytics Progress
 
IT Quality Testing and the Defect Management Process
IT Quality Testing and the Defect Management ProcessIT Quality Testing and the Defect Management Process
IT Quality Testing and the Defect Management ProcessYolanda Williams
 

Similar to Risk Based Testing in Agile context - Kees Blokland (20)

Fundamentals of testing
Fundamentals of testingFundamentals of testing
Fundamentals of testing
 
Risk Based Thinking ISO 9001 Presentation.pdf
Risk Based Thinking ISO 9001 Presentation.pdfRisk Based Thinking ISO 9001 Presentation.pdf
Risk Based Thinking ISO 9001 Presentation.pdf
 
EXTENT-2016: The Future of Software Testing
EXTENT-2016:	 The Future of Software TestingEXTENT-2016:	 The Future of Software Testing
EXTENT-2016: The Future of Software Testing
 
Implementing Risk Based Thinking in HLS OF ISO 9001:2015 - Praneet Surti
Implementing Risk Based Thinking in HLS OF ISO 9001:2015 - Praneet SurtiImplementing Risk Based Thinking in HLS OF ISO 9001:2015 - Praneet Surti
Implementing Risk Based Thinking in HLS OF ISO 9001:2015 - Praneet Surti
 
Industrial Training in Software Testing
Industrial Training in Software TestingIndustrial Training in Software Testing
Industrial Training in Software Testing
 
Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Inn...
Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Inn...Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Inn...
Neil Thompson - Value Inspired Testing: Renovating Risk-Based Testing and Inn...
 
Fundamentals of testing
Fundamentals of testingFundamentals of testing
Fundamentals of testing
 
Risk Driven Testing
Risk Driven TestingRisk Driven Testing
Risk Driven Testing
 
Risk Management
Risk Management Risk Management
Risk Management
 
UX Lead Product Management
UX Lead Product ManagementUX Lead Product Management
UX Lead Product Management
 
Fundamentals of testing (1)
Fundamentals of testing (1)Fundamentals of testing (1)
Fundamentals of testing (1)
 
TestHiveB
TestHiveBTestHiveB
TestHiveB
 
Bab 1
Bab 1Bab 1
Bab 1
 
Fundamentals of crowdsourced testing
Fundamentals of crowdsourced testingFundamentals of crowdsourced testing
Fundamentals of crowdsourced testing
 
Chapter 1 - Why Testing.ppt
Chapter 1 - Why Testing.pptChapter 1 - Why Testing.ppt
Chapter 1 - Why Testing.ppt
 
Presentasi fundamental of testing
Presentasi fundamental of testingPresentasi fundamental of testing
Presentasi fundamental of testing
 
18 Jul 2018 - FMEA and Risk Management in Practice
18 Jul 2018 - FMEA and Risk Management in Practice 18 Jul 2018 - FMEA and Risk Management in Practice
18 Jul 2018 - FMEA and Risk Management in Practice
 
Better Software East 2016: Evolving Automated to Continuous
Better Software East 2016: Evolving Automated to ContinuousBetter Software East 2016: Evolving Automated to Continuous
Better Software East 2016: Evolving Automated to Continuous
 
Navigating Your Product's Growth with Embedded Analytics
Navigating Your Product's Growth with Embedded Analytics Navigating Your Product's Growth with Embedded Analytics
Navigating Your Product's Growth with Embedded Analytics
 
IT Quality Testing and the Defect Management Process
IT Quality Testing and the Defect Management ProcessIT Quality Testing and the Defect Management Process
IT Quality Testing and the Defect Management Process
 

More from Kees Blokland

Introductie boek Testing Cloud Services.
Introductie boek Testing Cloud Services.Introductie boek Testing Cloud Services.
Introductie boek Testing Cloud Services.Kees Blokland
 
Introducing testing cloud services - Transformation to SaaS
Introducing testing cloud services - Transformation to SaaSIntroducing testing cloud services - Transformation to SaaS
Introducing testing cloud services - Transformation to SaaSKees Blokland
 
Product risico analyse in de praktijk (2010) - Kees Blokland
Product risico analyse in de praktijk (2010) - Kees BloklandProduct risico analyse in de praktijk (2010) - Kees Blokland
Product risico analyse in de praktijk (2010) - Kees BloklandKees Blokland
 
Monitoring en control op uitbestede testwerkzaamheden door Kees Blokland.
Monitoring en control op uitbestede testwerkzaamheden door Kees Blokland.Monitoring en control op uitbestede testwerkzaamheden door Kees Blokland.
Monitoring en control op uitbestede testwerkzaamheden door Kees Blokland.Kees Blokland
 
Mistakes Outsourcing Customers Make by Kees Blokland
Mistakes Outsourcing Customers Make by Kees BloklandMistakes Outsourcing Customers Make by Kees Blokland
Mistakes Outsourcing Customers Make by Kees BloklandKees Blokland
 
Metrics for (outsourced) test activities by Kees Blokland
Metrics for (outsourced) test activities by Kees BloklandMetrics for (outsourced) test activities by Kees Blokland
Metrics for (outsourced) test activities by Kees BloklandKees Blokland
 
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin PolNon-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin PolKees Blokland
 
Testing Cloud Services - Kees Blokland and Jeroen Mengerink
Testing Cloud Services - Kees Blokland and Jeroen MengerinkTesting Cloud Services - Kees Blokland and Jeroen Mengerink
Testing Cloud Services - Kees Blokland and Jeroen MengerinkKees Blokland
 
Useful stepping stones in growth towards Agile testing by Kees Blokland
Useful stepping stones in growth towards Agile testing by Kees BloklandUseful stepping stones in growth towards Agile testing by Kees Blokland
Useful stepping stones in growth towards Agile testing by Kees BloklandKees Blokland
 
Useful stepping stones in growth towards Agile testing door Kees Blokland.
Useful stepping stones in growth towards Agile testing door Kees Blokland.Useful stepping stones in growth towards Agile testing door Kees Blokland.
Useful stepping stones in growth towards Agile testing door Kees Blokland.Kees Blokland
 
EuroStar Webinar Testing Cloud Services Kees Blokland
EuroStar Webinar Testing Cloud Services Kees BloklandEuroStar Webinar Testing Cloud Services Kees Blokland
EuroStar Webinar Testing Cloud Services Kees BloklandKees Blokland
 

More from Kees Blokland (11)

Introductie boek Testing Cloud Services.
Introductie boek Testing Cloud Services.Introductie boek Testing Cloud Services.
Introductie boek Testing Cloud Services.
 
Introducing testing cloud services - Transformation to SaaS
Introducing testing cloud services - Transformation to SaaSIntroducing testing cloud services - Transformation to SaaS
Introducing testing cloud services - Transformation to SaaS
 
Product risico analyse in de praktijk (2010) - Kees Blokland
Product risico analyse in de praktijk (2010) - Kees BloklandProduct risico analyse in de praktijk (2010) - Kees Blokland
Product risico analyse in de praktijk (2010) - Kees Blokland
 
Monitoring en control op uitbestede testwerkzaamheden door Kees Blokland.
Monitoring en control op uitbestede testwerkzaamheden door Kees Blokland.Monitoring en control op uitbestede testwerkzaamheden door Kees Blokland.
Monitoring en control op uitbestede testwerkzaamheden door Kees Blokland.
 
Mistakes Outsourcing Customers Make by Kees Blokland
Mistakes Outsourcing Customers Make by Kees BloklandMistakes Outsourcing Customers Make by Kees Blokland
Mistakes Outsourcing Customers Make by Kees Blokland
 
Metrics for (outsourced) test activities by Kees Blokland
Metrics for (outsourced) test activities by Kees BloklandMetrics for (outsourced) test activities by Kees Blokland
Metrics for (outsourced) test activities by Kees Blokland
 
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin PolNon-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
 
Testing Cloud Services - Kees Blokland and Jeroen Mengerink
Testing Cloud Services - Kees Blokland and Jeroen MengerinkTesting Cloud Services - Kees Blokland and Jeroen Mengerink
Testing Cloud Services - Kees Blokland and Jeroen Mengerink
 
Useful stepping stones in growth towards Agile testing by Kees Blokland
Useful stepping stones in growth towards Agile testing by Kees BloklandUseful stepping stones in growth towards Agile testing by Kees Blokland
Useful stepping stones in growth towards Agile testing by Kees Blokland
 
Useful stepping stones in growth towards Agile testing door Kees Blokland.
Useful stepping stones in growth towards Agile testing door Kees Blokland.Useful stepping stones in growth towards Agile testing door Kees Blokland.
Useful stepping stones in growth towards Agile testing door Kees Blokland.
 
EuroStar Webinar Testing Cloud Services Kees Blokland
EuroStar Webinar Testing Cloud Services Kees BloklandEuroStar Webinar Testing Cloud Services Kees Blokland
EuroStar Webinar Testing Cloud Services Kees Blokland
 

Recently uploaded

Reliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesReliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesGlobus
 
Instrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowInstrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowGlobus
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Dmitry Zinoviev
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleAmir Moghimi
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxMindInventory
 
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...
CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...syedfaisal759877
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration TopicsGlobus
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with GlobusGlobus
 
Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCIOWomenMagazine
 
Building Research Applications with Globus PaaS
Building Research Applications with Globus PaaSBuilding Research Applications with Globus PaaS
Building Research Applications with Globus PaaSGlobus
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System AdministratorsGlobus
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfssuser82c38d
 
Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Fermin Galan
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flinkconfluent
 
Passbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentPassbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentThierry Gayet
 
Managing multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerManaging multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerThierry Gayet
 
Best Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusBest Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusGlobus
 
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio, Inc.
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureHironori Washizaki
 

Recently uploaded (20)

Reliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesReliable, Remote Computation at All Scales
Reliable, Remote Computation at All Scales
 
Instrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowInstrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a Flow
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scale
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptx
 
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...
CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration Topics
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with Globus
 
Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdf
 
Building Research Applications with Globus PaaS
Building Research Applications with Globus PaaSBuilding Research Applications with Globus PaaS
Building Research Applications with Globus PaaS
 
Globus for System Administrators
Globus for System AdministratorsGlobus for System Administrators
Globus for System Administrators
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdf
 
Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
Passbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentPassbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managment
 
Managing multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerManaging multicast/igmp stream on Docker
Managing multicast/igmp stream on Docker
 
Best Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusBest Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using Globus
 
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
 
2024 Trends Transforming Enterprise Resource Planning
2024 Trends Transforming Enterprise Resource Planning2024 Trends Transforming Enterprise Resource Planning
2024 Trends Transforming Enterprise Resource Planning
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about Architecture
 

Risk Based Testing in Agile context - Kees Blokland

  • 2. Risk based testing in Agile? context Huawei, Shenzhen 2016 Kees Blokland kees.blokland@polteq.com
  • 3. 3© 2016 Question to you: • What does risk based testing mean for you?
  • 4. 4© 2016 Testers must make choices What to test and what not to test What to test thoroughly, what to test superficially What to test first, what to test later What to test once, what to test each software build What to test by senior testers, what by junior testers What to test scripted, what to test exploratory, … just to name a few We cannot test all possible ways through the software
  • 5. 5© 2016 Testers must make choices What to test and what not to test What to test thoroughly, what to test superficially What to test first, what to test later What to test once, what to test each software build What to test by senior testers, what by junior testers What to test scripted, what to test exploratory, … just to name a few We base these choices on risk
  • 7. 7© 2016 Risk defined WIKIPEDIA The probability or threat of quantifiable damage, injury, liability, loss, or any other negative occurrence that is caused by external or internal vulnerabilities, and that may be avoided through pre-emptive action.
  • 8. 8© 2016 Risk defined Michael Bolton Some person will suffer loss, harm, annoyance, or reduced value because of a problem in the product that is triggered by some threat.
  • 9. 9© 2016 Risk defined WIKIPEDIA The probability or threat of quantifiable damage, injury, liability, loss, or any other negative occurrence that is caused by external or internal vulnerabilities, and that may be avoided through pre-emptive action. a system failure which is bad to a certain extend we don’t know for sure it will happen we may want to prevent it
  • 10. 10© 2016 BMW needs to call back 10.000 cars in China due to a software defect source: computable.nl Doors open spontaneously Gear switches from parked state and car starts rolling by itself from the hill
  • 11. 11© 2016 Huge traffic jam due to system failure source: computable.nl Huge traffic jams due to failing information boards
  • 13. 13© 2016 Testing: Test’s law on software failures R = L x IRisk Likelihood Impact NO RISK NO TEST
  • 14. 14© 2016 Risk assessment WIKIPEDIA The probability or threat of quantifiable damage, injury, liability, loss, or any other negative occurrence that is caused by external or internal vulnerabilities, and that may be avoided through pre-emptive action. What failures of the product could we anticipate. What impact would these have?
  • 15. 15© 2016 Risk assessment WIKIPEDIA The probability or threat of quantifiable damage, injury, liability, loss, or any other negative occurrence that is caused by external or internal vulnerabilities, and that may be avoided through pre-emptive action. What would cause faults in the product that make it fail like that? What is the likelihood of this to happen?
  • 16. 16© 2016 Risk reduction How can testing help to reduce the risks by finding these (types of) faults so preventing the failures to happen in the product?
  • 17. 17© 2016 break down break down recognizable by customer / user recognizable by development prioritized on business value acceptance criteria user scenario’s Feature refinement, break down
  • 18. 18© 2016 break down break down recognizable by customer / user recognizable by development prioritized on business value acceptance criteria user scenario’s Question • Discuss where to find these in your project
  • 19. 19© 2016 break down break down What failures could we anticipate. What impact would these have? What would cause faults in the product? What is the likelihood of this to happen? How can testing help to reduce the risks by finding faults causing the failures?
  • 20. 20© 2016 break down break down What failures could we anticipate. What impact would these have? Risk assessment Who represents the user / client / business?
  • 21. Risk Story Elements • Some PERSON(S) – a person, a group, the business, a bystander • will EXPERIENCE – bad feelings and a trigger for them • a PROBLEM – annoyance, loss, harm, diminished value… • with respect to SOMETHING DESIRABLE – capability, reliability, performance… • that CAN BE DETECTED – by an oracle (a means of recognizing a problem)… • in SOME SET OF CONDITIONS – perhaps invariably, perhaps intermittently… • because of a VULNERABILITY • a bug, a missing feature, an inconsistency… • in the SYSTEM. – some result, process, component, environment… © James Bach and Michael Bolton, 2016
  • 22. © James Bach and Michael Bolton, 2016 Some General Systems Risk Heuristics • A condition in the world that interacts with your product may happen rarely or frequently. • A problem may be detectable or undetectable at the time it first occurs or begins to occur (relative to our means of detection). • A problem may occur in a result, process, component, or environment. • In the realm of technology and society, problems fall into diverse categories: usability, performance, concurrency, legality, correctness, compatibility, etc. • A problem may impact a user, a bystander, or the business that produced the product. • A problem may cause inconvenience, annoyance, or harm. • …
  • 23. Examples • sometimes (in a given situation) one of the components OF THE PRODUCT visibly FAILS WITH RESPECT TO compatibility SUCH THAT bystander IS harmed • sometimes (in a given situation) some process OF THE PRODUCT visibly FAILS WITH RESPECT TO legality SUCH THAT business IS harmed • sometimes (in a given situation) some process OF THE PRODUCT visibly FAILS WITH RESPECT TO usability SUCH THAT user IS annoyed • always some result OF THE PRODUCT invisibly FAILS WITH RESPECT TO concurrency SUCH THAT bystander IS annoyed • always some process OF THE PRODUCT invisibly FAILS WITH RESPECT TO legality SUCH THAT bystander IS inconvenienced • always some result OF THE PRODUCT visibly FAILS WITH RESPECT TO performance SUCH THAT business IS annoyed • always some result OF THE PRODUCT visibly FAILS WITH RESPECT TO compatibility SUCH THAT bystander IS harmed • sometimes (in a given situation) some result OF THE PRODUCT invisibly FAILS WITH RESPECT TO correctness SUCH THAT business IS inconvenienced • always one of the components OF THE PRODUCT invisibly FAILS WITH RESPECT TO performance SUCH THAT user IS inconvenienced • always one of the components OF THE PRODUCT visibly FAILS WITH RESPECT TO correctness SUCH THAT user IS annoyed © James Bach and Michael Bolton, 2016
  • 24. 24© 2016 Question to you: what are your examples? and: who should we talk with?
  • 25. 25© 2016 Risk assessment – Discuss the impact Losing income Losing customers Societal Damage Claims by users Repair costs Damage to company image Compensation effort and cost Development and test cost Clients / Users / Business perspective
  • 26. 26© 2016 Risk assessment User story Risk Quality Impact User story 1 Template Unable to install client SW, loose customer functional H Customer does not understand how to operate (and terminates session) usability H User story 2 Upload & Preview Preview does not function (customer disappointed) functional M Unable to create a mailing, less customer value performance H Upload over 3G is taking too long (customer complains or stops) functional M User story 4 Date format incorrect functional L User story 5 Upload does not start functional H …
  • 27. 27© 2016 break down break down What would cause faults in the product? What is the likelihood of this to happen? Risk assessment
  • 28. 28© 2016 Risk Analysis Heuristics • Project factors – Things going on in projects, among people, may lead to bugs. • Technology factors – The structure and dynamics of technology itself may give rise to bugs. • Requirements factors – Aspects of requirements may indicate or promote the presence of bugs. • Operational factors – The circumstances and patterns of use affect the probability and impact of bugs. Risk Analysis Heuristics (for Digital Products) By James Bach and Michael Bolton v. 2.1
  • 29. 29© 2016 Example of technology factors logic platform Service boards Main NE linux lamps cabling subrack … Service Board-1 Main b power Service Board-n Service Board-2 … Feature map Customer application scenario 100G extra long haul 1550nm Protection switching Automatic laser shutdown Alarming AU4-4c channel provisioning Adding / removing wavelengths …. Software map Hardware mapNetwork map Service board Asic1 pwr Otherstuff Asic 2 Firmware optics NE NMS Here are the changes made
  • 30. 30© 2016 Risk assessment – another list for inspiration Frequency of use Inexperienced developers New tools High time pressure Complex functions Many changes Many interfaces Error prone functions
  • 31. 31© 2016 What are your examples?
  • 32. 32© 2016 Risk assessment User story 1: Template Q Risk Fault proba bility Unable to install client SW, loose customer F Install on windows; stable, predictable, hardly bugs seen before L F Install on macos; NEW, no experience, high time pressure H P Image becomes bigger, may cause performance problems M
  • 33. 33© 2016 Risk assessment User story 1: Template Q Risk Impact Fault prob. Risk class Unable to install client SW, loose customer F Install on windows; stable, predictable, hardly bugs seen before H L M F Install on macos; NEW, no experience, high time pressure H H H P Image becomes bigger, may cause performance problems H M M Impact H M L Fault pr. H H M L M M M L L M L L Risk class lookup These are most important!
  • 34. 34© 2016 Do you remember this slide?
  • 35. 35© 2016 Question to you: • How to test more thorough, more deeply?
  • 36. 36© 2016 How to map risks on tests? Techniques Superficial Implicit Thorough Specific Formal Multiple Model based Medium risk High risk Low risk Test levels Component Test + System Test + Acceptance Test + System Integr. T. Special test All Performance Security Usability Testers Junior Medior Senior Exploratory testing Free format Session based Session based by experts
  • 37. 37© 2016 How to map risks on tests? - continued Tested once Automated regr. test Test rounds Multiple Each Manual regression test Review - Informal Formal Specific Focus Code review - Pairing Pairing with testers Testers Junior Medior Senior Experts
  • 38. 38© 2016 Question to you: • What techniques do you know to do more deep testing?
  • 39. 39© 2016 Do you remember this slide?
  • 40. 40© 2016 What to test first and what later? In the end execute tests on low risk components / features / aspects First execute tests on high risk components / features / aspects Time Low severity defects undiscovered Most serious defects Low severity defects Then execute tests on medium risk components / features / aspects Less serious defects Project deadline
  • 41. 41© 2016 Determining testing strategy Backlog item list - Changes - Defects - Technical debt Items considered for this iteration Iteration, e.g. 3 weeks Iteration planning meeting and/or clarification / elaboration meeting Time  SE DE DV TE CR Example
  • 42. 42© 2016 Determining testing strategy Backlog item list - Changes - Defects - Technical debt Items considered for this iteration Iteration, e.g. 3 weeks Iteration planning meeting and/or clarification / elaboration meeting Time  SE DE DV TE CR Example • What ‘tools’ do we have? – Pair testing – Code review – Automated regression testing (/checking) – Manual regression testing – Unit testing – TDD – Exploratory sessions – Scripted testing (prepare for automated regression checks) – …
  • 43. 43© 2016 Reporting about testing (risk related) • What information have we gained about the quality of the product? – in general and specifically: – regarding the potential product failures identified in the risk assessment • What testing has been done so far? • What information do we have to determine further steps? – how testability of the product influenced testing – what further testing could be done (related to the risks) – the cost of further testing (effort and lead time) Any level: TE level, TPL level, TM level, …
  • 44. 44© 2016 Reporting about testing (risk related) • What information have we gained about the quality of the product? – in general and specifically: – regarding the potential product failures identified in the risk assessment • What testing has been done so far? • What information do we have to determine further steps? – how testability of the product influenced testing – what further testing could be done (related to the risks) – the cost of further testing (effort and lead time) Example 1: Software delivery delayed due to complexity problems SO: not all planned testing could be done AND: need EVEN MORE TESTING because delayed delivery increases product risk Example 2: Found many bugs so far, which are solved and retested BUT: more testing is likely to uncover more bugs SO: need MORE TESTING because many bugs found is is an indicator of high risk
  • 45. 45© 2016 Considerations on risk based testing • Risk based testing requires solid testing skills • Risk based testing requires dialogue with the client/user/customer • Risk based testing requires dialogue with developers • Information from testing can lower the risk level • Information from testing can ALSO indicate a HIGHER risk level
  • 46. 46© 2016 Last question to you: • Do YOU have questions?

Editor's Notes

  1. (C) Polteq IT Services BV
  2. (C) Polteq IT Services BV
  3. (C) Polteq IT Services BV
  4. (C) Polteq IT Services BV
  5. (C) Polteq IT Services BV
  6. (C) Polteq IT Services BV
  7. ‘mis niks’ insteek