SlideShare a Scribd company logo
1 of 38
Janet Gregory, DragonFire Inc.
Copyright 2015
StarWest 2015
Twitter id: janetgregoryca
It’s usually about bugs
the conversation about quality
Or perhaps it’s about traceability
– Did we test all the requirements
-- What is our code coverage?
“Managers who don’t
know how to measure
what they want, settle
for wanting what they
can measure.”
Russell Ackoff
What does all that
information really tell
you?
• Value
• Uncertainty
• Risk
• How might we really measure product quality?
Instead, let’s talk about:
• Whose job is it anyway?
• The mindset of the team
And perhaps a bit about:
7
For your value, which is better?
Which is right?
Can something be right?
But not meet a
customer’s needs?
Understand the value to the business!
How to You
Measure Value
Validate
Your
Learning
Risk and Uncertainty
14
Copyright 2015 : Janet Gregory – DragonFire Inc.
Uncertainty
◦ not known or definite
◦ not completely confident or sure of something
17
Liz Keogh – lizkeogh.com/embracing-uncertainty/
Risk
• a situation involving exposure to danger
• expose someone (or something valued) to danger
• possibility of something unpleasant happening
There are 2 kinds of risks
1. Project risk
2. Product risk
With permission from Dan North - a Software Faster pattern
With permission from Dan North - a Software Faster pattern
With permission from Dan North - a Software Faster pattern
• Project, product, organization, reputation,
contractual, regulatory, legal (ex. privacy),
ethical …..
• ASK – Why do your
customers want you
to test for them?
Visualize
Involve your whole team
including customers,
in the talk about risks & testing
But don’t
forget the
big picture
Look at the
details
Risk and Uncertainty
Is it your job?
What about the Who???
If every team member took a holistic view of
the product release and the risks it faces, then
every person on the team would be a critical
contributor to the delivery of that solution.
Matt Mansell - paraphrased
Is it the tester’s job
to think about quality?
So, do you follow behind looking for bugs?
or …. are you looking ahead for the risks?
29
Can we really
measure product
quality …..
if we don’t measure:
- bugs, or
- requirements or
- coverage?
• Customer satisfaction?
• Customer retention?
• The type of issues found in production?
Use analytics
• Who uses your product?
• Do they use it in the way you expected?
• Perhaps we start with a risk
profile, and focus on a risk-based
solution delivery.
Know your customers,
your stakeholders
Manage your uncertainty
Mitigate your risk
Understand the problem
Understand the value
from bugs and
traceability
to business value, risks
and uncertainty
about
product quality
From finding and
counting bugs
To preventing bugs
and measuring value
about measuring
product quality
Agile Testing: A Practical Guide for Testers and Agile Teams
More Agile Testing: Learning Journeys for the Whole Team
By Janet Gregory and Lisa Crispin
www.agiletester.ca
www.agiletester.com
Contact info
www.janetgregory.ca
Email: janet@agiletester.ca
Twitter: janetgregoryca
37
Copyright 2015 : Janet Gregory – DragonFire Inc.
References
• Dan North, StarEast 2015 keynote
• http://lizkeogh.com/embracing-uncertainty/
• Dave Snowden on Cynafin
http://www.youtube.com/watch?v=N7oz366X0-8
• Matt Mansell – Risk-based testing, ANZTB 2014
http://www.anztb.org/userfiles/files/MATT_MANSELL_EffectiveRiskB
asedTesting-Distribution.pdf
• Rob Lambert http://thesocialtester.co.uk/how-do-you-measure-the-
effectiveness-of-a-tester-the-only-calculation-you-need/
• http://cherylquirion.com/
• www.lisacrispin.com

More Related Content

What's hot

Get testing bottlenecks out of your pipelines
Get testing bottlenecks out of your pipelinesGet testing bottlenecks out of your pipelines
Get testing bottlenecks out of your pipelineslisacrispin
 
Agile Testing in the Enterprise
Agile Testing in the Enterprise Agile Testing in the Enterprise
Agile Testing in the Enterprise Janet Gregory
 
Creating Agile Test Strategies for Larger Enterprises
Creating Agile Test Strategies for Larger EnterprisesCreating Agile Test Strategies for Larger Enterprises
Creating Agile Test Strategies for Larger EnterprisesTEST Huddle
 
Cherrypic 2016-agile-testing
Cherrypic 2016-agile-testingCherrypic 2016-agile-testing
Cherrypic 2016-agile-testingLuis Franqueira
 
Engineering Speed
Engineering SpeedEngineering Speed
Engineering SpeedTerry Xu
 
Why agile testing isn't working
Why agile testing isn't workingWhy agile testing isn't working
Why agile testing isn't workingXebia Nederland BV
 
KAA How to get your Good agile teams to Great
KAA How to get your Good agile teams to GreatKAA How to get your Good agile teams to Great
KAA How to get your Good agile teams to GreatDavid Hawks
 
Dr. house would be a great product management
Dr. house would be a great product managementDr. house would be a great product management
Dr. house would be a great product managementTautvydas Gylys
 
Lessons about experiments
Lessons about experiments Lessons about experiments
Lessons about experiments Tautvydas Gylys
 
Accelerating Product Delivery with Design Sprints
Accelerating Product Delivery with Design SprintsAccelerating Product Delivery with Design Sprints
Accelerating Product Delivery with Design SprintsAnthony Broad-Crawford
 
Automation strategy step by step
Automation strategy step by stepAutomation strategy step by step
Automation strategy step by stepYevheniia Hlovatska
 
ЄВГЕНІЯ ГЛОВАЦЬКА «Automation Strategy Step by Step» Online QADay 2021
ЄВГЕНІЯ ГЛОВАЦЬКА «Automation Strategy Step by Step» Online QADay 2021ЄВГЕНІЯ ГЛОВАЦЬКА «Automation Strategy Step by Step» Online QADay 2021
ЄВГЕНІЯ ГЛОВАЦЬКА «Automation Strategy Step by Step» Online QADay 2021QADay
 
Test Strategy-The real silver bullet in testing by Matthew Eakin
Test Strategy-The real silver bullet in testing by Matthew EakinTest Strategy-The real silver bullet in testing by Matthew Eakin
Test Strategy-The real silver bullet in testing by Matthew EakinQA or the Highway
 
Testing is Not a 9 to 5 Job - talk by industry executive Mike Lyles
Testing is Not a 9 to 5 Job - talk by industry executive Mike LylesTesting is Not a 9 to 5 Job - talk by industry executive Mike Lyles
Testing is Not a 9 to 5 Job - talk by industry executive Mike LylesApplitools
 
Building Lean and Agile in the Real World
Building Lean and Agile in the Real WorldBuilding Lean and Agile in the Real World
Building Lean and Agile in the Real WorldKevin Goldsmith
 
5 why fishbone
5 why fishbone5 why fishbone
5 why fishboneAdi Shukla
 
Testing in a Continuous World
Testing in a Continuous WorldTesting in a Continuous World
Testing in a Continuous WorldLisi Hocke
 
WEBINAR: How to Flip the Conventional Lean Six Sigma Classroom Approach and G...
WEBINAR: How to Flip the Conventional Lean Six Sigma Classroom Approach and G...WEBINAR: How to Flip the Conventional Lean Six Sigma Classroom Approach and G...
WEBINAR: How to Flip the Conventional Lean Six Sigma Classroom Approach and G...GoLeanSixSigma.com
 

What's hot (20)

Get testing bottlenecks out of your pipelines
Get testing bottlenecks out of your pipelinesGet testing bottlenecks out of your pipelines
Get testing bottlenecks out of your pipelines
 
Agile Testing in the Enterprise
Agile Testing in the Enterprise Agile Testing in the Enterprise
Agile Testing in the Enterprise
 
Creating Agile Test Strategies for Larger Enterprises
Creating Agile Test Strategies for Larger EnterprisesCreating Agile Test Strategies for Larger Enterprises
Creating Agile Test Strategies for Larger Enterprises
 
A3 Process intro
A3 Process introA3 Process intro
A3 Process intro
 
Cherrypic 2016-agile-testing
Cherrypic 2016-agile-testingCherrypic 2016-agile-testing
Cherrypic 2016-agile-testing
 
Engineering Speed
Engineering SpeedEngineering Speed
Engineering Speed
 
Why agile testing isn't working
Why agile testing isn't workingWhy agile testing isn't working
Why agile testing isn't working
 
KAA How to get your Good agile teams to Great
KAA How to get your Good agile teams to GreatKAA How to get your Good agile teams to Great
KAA How to get your Good agile teams to Great
 
Dr. house would be a great product management
Dr. house would be a great product managementDr. house would be a great product management
Dr. house would be a great product management
 
Lessons about experiments
Lessons about experiments Lessons about experiments
Lessons about experiments
 
Accelerating Product Delivery with Design Sprints
Accelerating Product Delivery with Design SprintsAccelerating Product Delivery with Design Sprints
Accelerating Product Delivery with Design Sprints
 
Automation strategy step by step
Automation strategy step by stepAutomation strategy step by step
Automation strategy step by step
 
WEBINAR: Introduction to PDCA
WEBINAR: Introduction to PDCAWEBINAR: Introduction to PDCA
WEBINAR: Introduction to PDCA
 
ЄВГЕНІЯ ГЛОВАЦЬКА «Automation Strategy Step by Step» Online QADay 2021
ЄВГЕНІЯ ГЛОВАЦЬКА «Automation Strategy Step by Step» Online QADay 2021ЄВГЕНІЯ ГЛОВАЦЬКА «Automation Strategy Step by Step» Online QADay 2021
ЄВГЕНІЯ ГЛОВАЦЬКА «Automation Strategy Step by Step» Online QADay 2021
 
Test Strategy-The real silver bullet in testing by Matthew Eakin
Test Strategy-The real silver bullet in testing by Matthew EakinTest Strategy-The real silver bullet in testing by Matthew Eakin
Test Strategy-The real silver bullet in testing by Matthew Eakin
 
Testing is Not a 9 to 5 Job - talk by industry executive Mike Lyles
Testing is Not a 9 to 5 Job - talk by industry executive Mike LylesTesting is Not a 9 to 5 Job - talk by industry executive Mike Lyles
Testing is Not a 9 to 5 Job - talk by industry executive Mike Lyles
 
Building Lean and Agile in the Real World
Building Lean and Agile in the Real WorldBuilding Lean and Agile in the Real World
Building Lean and Agile in the Real World
 
5 why fishbone
5 why fishbone5 why fishbone
5 why fishbone
 
Testing in a Continuous World
Testing in a Continuous WorldTesting in a Continuous World
Testing in a Continuous World
 
WEBINAR: How to Flip the Conventional Lean Six Sigma Classroom Approach and G...
WEBINAR: How to Flip the Conventional Lean Six Sigma Classroom Approach and G...WEBINAR: How to Flip the Conventional Lean Six Sigma Classroom Approach and G...
WEBINAR: How to Flip the Conventional Lean Six Sigma Classroom Approach and G...
 

Similar to Change the conversation keynote StarWest 2015

Testing in modern times a story about quality and value - agile testing dev ...
Testing in modern times  a story about quality and value - agile testing dev ...Testing in modern times  a story about quality and value - agile testing dev ...
Testing in modern times a story about quality and value - agile testing dev ...Huib Schoots
 
GDG Cloud Southlake #5 Eric Harvieux: Site Reliability Engineering (SRE) in P...
GDG Cloud Southlake #5 Eric Harvieux: Site Reliability Engineering (SRE) in P...GDG Cloud Southlake #5 Eric Harvieux: Site Reliability Engineering (SRE) in P...
GDG Cloud Southlake #5 Eric Harvieux: Site Reliability Engineering (SRE) in P...James Anderson
 
Good agile / Bad agile: Proving the value of Agile to a skeptical organization
Good agile / Bad agile: Proving the value of Agile to a skeptical organizationGood agile / Bad agile: Proving the value of Agile to a skeptical organization
Good agile / Bad agile: Proving the value of Agile to a skeptical organizationAlan Albert
 
VSL Las Vegas 2023 - Measuring Up! How To Choose Agile Metrics
VSL Las Vegas 2023 - Measuring Up! How To Choose Agile MetricsVSL Las Vegas 2023 - Measuring Up! How To Choose Agile Metrics
VSL Las Vegas 2023 - Measuring Up! How To Choose Agile MetricsAngela Dugan
 
Presented at Ford's 2017 Global IT Learning Summit (GLITS)
Presented at Ford's 2017 Global IT Learning Summit (GLITS)Presented at Ford's 2017 Global IT Learning Summit (GLITS)
Presented at Ford's 2017 Global IT Learning Summit (GLITS)Ron Lazaro
 
Id camp x dicoding live : persiapan jadi software engineer hebat 101
Id camp x dicoding live : persiapan jadi software engineer hebat 101Id camp x dicoding live : persiapan jadi software engineer hebat 101
Id camp x dicoding live : persiapan jadi software engineer hebat 101DicodingEvent
 
Growing a Culture of Quality
Growing a Culture of QualityGrowing a Culture of Quality
Growing a Culture of QualitySimon Prior
 
User Experience Measurement and Analysis: Perception Testing
User Experience Measurement and Analysis: Perception TestingUser Experience Measurement and Analysis: Perception Testing
User Experience Measurement and Analysis: Perception TestingElementive
 
LTK - FC - Supply Chain - Startup Challenge v3.pdf
LTK - FC - Supply Chain - Startup Challenge v3.pdfLTK - FC - Supply Chain - Startup Challenge v3.pdf
LTK - FC - Supply Chain - Startup Challenge v3.pdfjeroen_tjepkema
 
How to Run Product Discovery Experiments in FinTech
How to Run Product Discovery Experiments in FinTechHow to Run Product Discovery Experiments in FinTech
How to Run Product Discovery Experiments in FinTechProduct School
 
October 2018 Agile Connect Lisbon Meetup
 October 2018 Agile Connect Lisbon Meetup October 2018 Agile Connect Lisbon Meetup
October 2018 Agile Connect Lisbon MeetupAgile Connect®
 
Huib Schoots Testing in modern times - a story about Quality and Value - Test...
Huib Schoots Testing in modern times - a story about Quality and Value - Test...Huib Schoots Testing in modern times - a story about Quality and Value - Test...
Huib Schoots Testing in modern times - a story about Quality and Value - Test...FiSTB
 
Rev Up Your Lead Engine With Predictive Scoring
Rev Up Your Lead Engine With Predictive ScoringRev Up Your Lead Engine With Predictive Scoring
Rev Up Your Lead Engine With Predictive ScoringMarketo
 
How do we fix testing
How do we fix testingHow do we fix testing
How do we fix testingPeter Varhol
 
Global Talent Acquisition: Metrics that Matter and Metrics that Mean NOTHING
Global Talent Acquisition: Metrics that Matter and Metrics that Mean NOTHINGGlobal Talent Acquisition: Metrics that Matter and Metrics that Mean NOTHING
Global Talent Acquisition: Metrics that Matter and Metrics that Mean NOTHINGRecruitingDaily.com LLC
 
Empowering Outcome-Driven Teams by Ibotta PM
Empowering Outcome-Driven Teams by Ibotta PMEmpowering Outcome-Driven Teams by Ibotta PM
Empowering Outcome-Driven Teams by Ibotta PMProduct School
 
Lean Startup for Project Managers
Lean Startup for Project ManagersLean Startup for Project Managers
Lean Startup for Project ManagersCaroline Gordon
 
Estimation tricks and traps
Estimation tricks and trapsEstimation tricks and traps
Estimation tricks and trapsMarta Kossowska
 

Similar to Change the conversation keynote StarWest 2015 (20)

Testing in modern times a story about quality and value - agile testing dev ...
Testing in modern times  a story about quality and value - agile testing dev ...Testing in modern times  a story about quality and value - agile testing dev ...
Testing in modern times a story about quality and value - agile testing dev ...
 
GDG Cloud Southlake #5 Eric Harvieux: Site Reliability Engineering (SRE) in P...
GDG Cloud Southlake #5 Eric Harvieux: Site Reliability Engineering (SRE) in P...GDG Cloud Southlake #5 Eric Harvieux: Site Reliability Engineering (SRE) in P...
GDG Cloud Southlake #5 Eric Harvieux: Site Reliability Engineering (SRE) in P...
 
Good agile / Bad agile: Proving the value of Agile to a skeptical organization
Good agile / Bad agile: Proving the value of Agile to a skeptical organizationGood agile / Bad agile: Proving the value of Agile to a skeptical organization
Good agile / Bad agile: Proving the value of Agile to a skeptical organization
 
VSL Las Vegas 2023 - Measuring Up! How To Choose Agile Metrics
VSL Las Vegas 2023 - Measuring Up! How To Choose Agile MetricsVSL Las Vegas 2023 - Measuring Up! How To Choose Agile Metrics
VSL Las Vegas 2023 - Measuring Up! How To Choose Agile Metrics
 
Presented at Ford's 2017 Global IT Learning Summit (GLITS)
Presented at Ford's 2017 Global IT Learning Summit (GLITS)Presented at Ford's 2017 Global IT Learning Summit (GLITS)
Presented at Ford's 2017 Global IT Learning Summit (GLITS)
 
Id camp x dicoding live : persiapan jadi software engineer hebat 101
Id camp x dicoding live : persiapan jadi software engineer hebat 101Id camp x dicoding live : persiapan jadi software engineer hebat 101
Id camp x dicoding live : persiapan jadi software engineer hebat 101
 
Growing a Culture of Quality
Growing a Culture of QualityGrowing a Culture of Quality
Growing a Culture of Quality
 
User Experience Measurement and Analysis: Perception Testing
User Experience Measurement and Analysis: Perception TestingUser Experience Measurement and Analysis: Perception Testing
User Experience Measurement and Analysis: Perception Testing
 
LTK - FC - Supply Chain - Startup Challenge v3.pdf
LTK - FC - Supply Chain - Startup Challenge v3.pdfLTK - FC - Supply Chain - Startup Challenge v3.pdf
LTK - FC - Supply Chain - Startup Challenge v3.pdf
 
How to Run Product Discovery Experiments in FinTech
How to Run Product Discovery Experiments in FinTechHow to Run Product Discovery Experiments in FinTech
How to Run Product Discovery Experiments in FinTech
 
October 2018 Agile Connect Lisbon Meetup
 October 2018 Agile Connect Lisbon Meetup October 2018 Agile Connect Lisbon Meetup
October 2018 Agile Connect Lisbon Meetup
 
Huib Schoots Testing in modern times - a story about Quality and Value - Test...
Huib Schoots Testing in modern times - a story about Quality and Value - Test...Huib Schoots Testing in modern times - a story about Quality and Value - Test...
Huib Schoots Testing in modern times - a story about Quality and Value - Test...
 
Rev Up Your Lead Engine With Predictive Scoring
Rev Up Your Lead Engine With Predictive ScoringRev Up Your Lead Engine With Predictive Scoring
Rev Up Your Lead Engine With Predictive Scoring
 
Rover Orientation
Rover OrientationRover Orientation
Rover Orientation
 
How do we fix testing
How do we fix testingHow do we fix testing
How do we fix testing
 
Nasty Impediments: Unclog the Pipe for Business Agility
Nasty Impediments: Unclog the Pipe for Business AgilityNasty Impediments: Unclog the Pipe for Business Agility
Nasty Impediments: Unclog the Pipe for Business Agility
 
Global Talent Acquisition: Metrics that Matter and Metrics that Mean NOTHING
Global Talent Acquisition: Metrics that Matter and Metrics that Mean NOTHINGGlobal Talent Acquisition: Metrics that Matter and Metrics that Mean NOTHING
Global Talent Acquisition: Metrics that Matter and Metrics that Mean NOTHING
 
Empowering Outcome-Driven Teams by Ibotta PM
Empowering Outcome-Driven Teams by Ibotta PMEmpowering Outcome-Driven Teams by Ibotta PM
Empowering Outcome-Driven Teams by Ibotta PM
 
Lean Startup for Project Managers
Lean Startup for Project ManagersLean Startup for Project Managers
Lean Startup for Project Managers
 
Estimation tricks and traps
Estimation tricks and trapsEstimation tricks and traps
Estimation tricks and traps
 

Recently uploaded

What need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersWhat need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersEmilyJiang23
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAlluxio, Inc.
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAlluxio, Inc.
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Krakówbim.edu.pl
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems ApproachNeo4j
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...rajkumar669520
 
Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024Soroosh Khodami
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024Shane Coughlan
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...naitiksharma1124
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Andrea Goulet
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfDeskTrack
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAlluxio, Inc.
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Gáspár Nagy
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...Alluxio, Inc.
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024vaibhav130304
 
How to pick right visual testing tool.pdf
How to pick right visual testing tool.pdfHow to pick right visual testing tool.pdf
How to pick right visual testing tool.pdfTestgrid.io
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesNeo4j
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionWave PLM
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)Max Lee
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfMehmet Akar
 

Recently uploaded (20)

What need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersWhat need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java Developers
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning Framework
 
Agnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in KrakówAgnieszka Andrzejewska - BIM School Course in Kraków
Agnieszka Andrzejewska - BIM School Course in Kraków
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
 
Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in Michelangelo
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024
 
How to pick right visual testing tool.pdf
How to pick right visual testing tool.pdfHow to pick right visual testing tool.pdf
How to pick right visual testing tool.pdf
 
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesGraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product Updates
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion Production
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdf
 

Change the conversation keynote StarWest 2015

  • 1. Janet Gregory, DragonFire Inc. Copyright 2015 StarWest 2015 Twitter id: janetgregoryca
  • 2. It’s usually about bugs the conversation about quality
  • 3.
  • 4. Or perhaps it’s about traceability – Did we test all the requirements -- What is our code coverage?
  • 5. “Managers who don’t know how to measure what they want, settle for wanting what they can measure.” Russell Ackoff
  • 6. What does all that information really tell you?
  • 7. • Value • Uncertainty • Risk • How might we really measure product quality? Instead, let’s talk about: • Whose job is it anyway? • The mindset of the team And perhaps a bit about: 7
  • 8.
  • 9. For your value, which is better? Which is right?
  • 10. Can something be right? But not meet a customer’s needs?
  • 11. Understand the value to the business!
  • 12.
  • 13. How to You Measure Value Validate Your Learning
  • 14. Risk and Uncertainty 14 Copyright 2015 : Janet Gregory – DragonFire Inc.
  • 15. Uncertainty ◦ not known or definite ◦ not completely confident or sure of something
  • 16.
  • 17. 17 Liz Keogh – lizkeogh.com/embracing-uncertainty/
  • 18. Risk • a situation involving exposure to danger • expose someone (or something valued) to danger • possibility of something unpleasant happening
  • 19. There are 2 kinds of risks 1. Project risk 2. Product risk
  • 20. With permission from Dan North - a Software Faster pattern
  • 21. With permission from Dan North - a Software Faster pattern
  • 22. With permission from Dan North - a Software Faster pattern
  • 23. • Project, product, organization, reputation, contractual, regulatory, legal (ex. privacy), ethical ….. • ASK – Why do your customers want you to test for them?
  • 25. Involve your whole team including customers, in the talk about risks & testing
  • 26. But don’t forget the big picture Look at the details
  • 28. Is it your job? What about the Who??? If every team member took a holistic view of the product release and the risks it faces, then every person on the team would be a critical contributor to the delivery of that solution. Matt Mansell - paraphrased Is it the tester’s job to think about quality?
  • 29. So, do you follow behind looking for bugs? or …. are you looking ahead for the risks? 29
  • 30. Can we really measure product quality ….. if we don’t measure: - bugs, or - requirements or - coverage?
  • 31. • Customer satisfaction? • Customer retention? • The type of issues found in production?
  • 32. Use analytics • Who uses your product? • Do they use it in the way you expected?
  • 33. • Perhaps we start with a risk profile, and focus on a risk-based solution delivery.
  • 34. Know your customers, your stakeholders Manage your uncertainty Mitigate your risk Understand the problem Understand the value
  • 35. from bugs and traceability to business value, risks and uncertainty about product quality
  • 36. From finding and counting bugs To preventing bugs and measuring value about measuring product quality
  • 37. Agile Testing: A Practical Guide for Testers and Agile Teams More Agile Testing: Learning Journeys for the Whole Team By Janet Gregory and Lisa Crispin www.agiletester.ca www.agiletester.com Contact info www.janetgregory.ca Email: janet@agiletester.ca Twitter: janetgregoryca 37 Copyright 2015 : Janet Gregory – DragonFire Inc.
  • 38. References • Dan North, StarEast 2015 keynote • http://lizkeogh.com/embracing-uncertainty/ • Dave Snowden on Cynafin http://www.youtube.com/watch?v=N7oz366X0-8 • Matt Mansell – Risk-based testing, ANZTB 2014 http://www.anztb.org/userfiles/files/MATT_MANSELL_EffectiveRiskB asedTesting-Distribution.pdf • Rob Lambert http://thesocialtester.co.uk/how-do-you-measure-the- effectiveness-of-a-tester-the-only-calculation-you-need/ • http://cherylquirion.com/ • www.lisacrispin.com

Editor's Notes

  1. I want you to think about your last release to production – to your customers. Was it successful? <show of hands>
  2. The conversation I am talking about is the conversation on quality How many bugs …. What kind of bugs… and so on. Only matters when there are too many to be able to use the application And that’s really not what is important to the customer – in today’s market, if you are paying for something, you expect a quality product. What’s important to your customer?
  3. We measure … What type of bugs, What severity, what priority We have defect triage – estimating how long to fix Waste
  4. Sometimes we measure other things, like ‘did we get all our requirements in, and did we test them?” How important is this? In today’s age of agile, a disciplined delivery team works at a feature / story together with the customer, who accepts. We know we test all the requirements because we do it together. There’s not usually a need to add extra work to trace. If necessary, we can extract the tests and test results from the automation for audits. There are exceptions, but I don’t come across them very often.
  5. Measuring anything gives them - management a feeling of control, but it’s an illusion. We can’t control.
  6. Probably not much … or as much as you think. Much of the metrics we use, probably doesn’t‘ even tell us about the quality of the product, but more about the quality of the process. Does the customer care about the process or the product. Delivery teams often get stuck in the details.
  7. Value – what is it? Uncertainty – how can we reduce it? Risk – how do we mitigate it? How might we really measure product quality
  8. Carcassonne, France – medieval city Can we deliver what they want …. Or better yet, what they need?
  9. value is not the same for everyone When we understand what problem the customer really has, maybe we’ll stand a chance of actually delivering what they need.
  10. So instead of specifying # of steps, the rise of each step, etc. Maybe we start with “We need to be able to escape from the floor we are on, and get either up to the roof, or down to the street”. When we build it, we have a common understanding of the problem we are trying to solve, when we test it, we can test to the problem & solution, not only to the specification.
  11. When we think about business value, we need to think about priorities. Priorities… the most valuable first Let’s think about it.. Can we break a feature into smaller bits and deliver “just enough” Story done Feature done ... Work with customers to understand the importance of acceptance. Try to get small acceptance of features – define feature acceptance, not only story acceptance
  12. I realized that I had done the pieces I really wanted to .. The dress, the pajamas, those are really boring – lots of white, and not much value to me.
  13. Think “outside the box” – it doesn’t have to be major, and sometimes one new idea makes a process a whole lot easier. The most successful teams enjoy a culture of learning where everyone on the team is free to raise issues and experiment. Experiment to change how you think about measuring value. Validate early – Customer acceptance – on each story, on each feature. Reduce the risk of getting it wrong.
  14. Project managers often want certainty, management wants control – or the illusion of control But perhaps, we shouldn’t aim for control, but think about how to reduce uncertainty –
  15. Introduce the Cynefin model so people can go explore and learn. Cynefin Framework (pronounced Ki neh’ vin) is about different domains/contexts -------a sense making model (Dave Snowden Simple (obvious/known)– we’ve done it before; example – communication protocols Complicated – harder, but not new Complex – unknown Chaotic – reacting, responding to fires
  16. Models can help us choose how to tackle a problem … Is it simple – we’ve done it before : the same, no need for analysis Complicated – harder, but not new - can apply good practices like break into smaller chunks, requires expertise, good practices like ATDD or BDD Complex – unknown - needs probing, questioning, works through examples to understand, experiment, get a better understanding of what it means before we commit Chaotic – reacting, responding to fires - couldn’t treat each problem exactly the same. We don’t want to do the same thing for every single new feature that comes up. We need to think about how to reduce the uncertainty in those complex problem areas. But not waste time on known problem areas.
  17. Do we have danger in our s/w development - Especially if we are working on a life critical application. Comes back to context.
  18. Example of project risk – scope, schedule, budget Example of product risk – features, quality attributes such as performance From a testing perspective, we usually mean the second – risks to the product.
  19. In Dan North’s keynote at StarEast, he talked about several things including the idea of risk planes. One of his Software Faster patterns - These are his pictures – used with his permission. Risk planes – 3 dimensions: likelihood, impact and probability. D – high impact, high likelihood should probably have higher coverage … F – unlikely to fail, and if it does fail, do we care? 80% too much testing We call this risk based testing,
  20. Components work together, not in isolation. User journeys…. Weakest link
  21. Each context belongs to a human being Testers need to be inside each of the stakeholders heads, and ask… We go back to the value to the customer, what is their risk threshold – what can they live with.
  22. So, I said there 2 risks – but that is a myth … There are many stakeholders, many risks. Their perceived risk may not be what you are thinking at all. By asking why they want you to test, perhaps you can figure out what the risks are they are trying to mitigate – what is the most important thing? Ask not only what’s the best, but what’s the worst thing that can happen. Unnecessary features add unnecessary risks. Story about Australia – study showed that speed and alcohol were not the only major causes of accidents. It turns out that tiredness is right up there as well, so they addressed that issue by putting reminder signs like this along the way, and made sure there were rest stops on the major highways. The accidents were reduced. Started in 1986 How do we make sure our team and our stakeholders get the same understanding.
  23. Risk is reduced when more people know the issues … It’s about transparency – Make it easy for people to participate in helping …
  24. Use brain storming techniques such as mind mapping This is about testing early - Think about prevent defects instead of counting them after. How do you know when you prevented a defect? By the ah ha someone says. But that’s harder to prove… harder to count. When we make testing visible and reducing misinterpretations, and misunderstanding – it helps to building the right thing. Think about the business rules and explore examples - at all levels
  25. We often get engrossed in the details – that’s where our expertise comes in, but we need to think about the bigger picture. We need to step back a bit and look at where will it grow, how big will it get, do I need …..
  26. I liked the way Matt Mansell really says Quality is everyone’s job. A contributor the solution – much better than the person who breaks the software… Holistic view – everyone’s responsibility ;
  27. Do you look at your problems and really understand them. I see so many teams that aren’t addressing real problems, or don’t understand some basic ideas.
  28. Customer validation and experimentation needs to continue throughout product/solution development - from the initial seed of an idea through to feature enhancements of a mature product/solution. I like to think of it in terms – - Problem validation, - making sure we understand the problem we are trying to solve - solution (definition) validation – making sure we have a shared understanding of what we are trying to build – testing assumptions early - implementation validation…(testing the product) Our attention should be on solution-market fit versus number of bugs we found. Right now, if feels like we are often victims of our rituals – what we know, what we are comfortable with.
  29. They are good things. Is it enough? If you aim for zero tolerance during development - zero known bugs escaping an iteration (or story or feature), then we can look very closely at the issues found by the customer because there are few. What about the business value the software adds? What about revenue generated by the the software added? What about the cycle time of work? --- again – that’s measuring process quality – not product quality.
  30. Monitor how your customers use your product. How is the customer impacted in real time so we can act quickly.
  31. If we address all the high risk items and only low risk items at the end of the project tells you what? For example, if one of the biggest risks is usability, perhaps we can use some testing techniques borrowed from the user experience world – techniques like customer touch points
  32. Use different approaches and tools Ex. Story mapping, Impact mapping, Structured conversations
  33. How can we become more flexible in our thinking – in getting better at what we give to our customers. Just imagine, putting out a release with very few known defects … and being able to measure customer satisfaction not by bugs, but by real value. We know that you get what you measure – think what is important to your organization. Stop, and look at what you are measuring. Does it add value to the organization? Is there something else that we could measure, or change. If you stop counting your defects, (which includes, logging them all and triaging), how much time would that give you to start thinking about the business problem, and being able to participate in validating the solution to that problem before the code is even written. Knowing how many bugs are in your software tells you … how bad it is, and I’m pretty sure that is not what organization thinks is important. We need to create a good / create impact on the minds of the customers. That's is what will give loyalty to your product.
  34. 37