SlideShare a Scribd company logo
1 of 39
Download to read offline
Continuous
Improvement
  How rapid release cycles
    alter QA and testing.


       Noah Sussman
DevopsDay Mountain View, 2012
       @noahsussman
        #devopsdays
The Canonical Agile
    Release Cycle
Sprints of two weeks or more in length.
Start deployment process at the end of the sprint.
QA is part of the deployment process.
QA must be complete before new code goes live.
The Continuous
       Release Cycle
Minimum viable feature set.
Deployment is decoupled from release.
Real-time data on how releases impact revenue.
Constant tweaks to live features.
Releasing a feature
is decoupled from
deploying code.




    David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122
An airport without
an air traffic controller.
         —Chad Dickerson
This Part Really Is
Different From Agile
Large features are deployed piecemeal over time.
Every feature is part of an A/B campaign.
Dark and Admin-Only launches.
Wire-Offs and Config Flags.
There is no “Done Done.”
Observed Behavior
Of Complex Systems
Emergent behaviors require unplanned responses.
Improvements, too are discovered not designed.
Users of the system have complex expectations.
Such systems are never “complete.”
QA Happens When?
First of all, what is “Quality Assurance?”
Authoritatively assuring that there are no defects?
That’s impossible.
Testing is everyone’s job.
Myths About Bug
       Detection
There are a finite number of bugs.
There are a finite number of detectable bugs.
All severity one bugs can be found before release
Software is built to specifications.
At some point, software is finished.
The Biggest Myth
Bugs have complex, unpredictable causes.


In fact, most errors in software are the results of
incorrect assumptions made by programmers.
Many Small
Anomalies Combined
The “Swiss Cheese” model of risk presents us
with a strong case for prioritizing the elimination of
small errors rather than focusing on the mitigation
of large catastrophic failures.


Unit testing is great at eliminating small errors.
The whole time I’m
programming, I’m
constantly checking
my assumptions.
   —Rasmus Lerdorf
Resilience, Not
          “Quality”
Readable code.
Reasonable test coverage.
Sane architecture.
Good debugging tools.
An engineering culture that values refactoring.
These are measureable goals.
Manual Testing
It doesn’t always look like you think it looks.


Real-Time Monitoring is the new face of testing.
Anomaly detection is hard.
Watching The Graphs
Etsy collects well over a quarter million metrics.
Deciding which ones matter is a human problem.
Everyone watches some subset of the graphs.
Human vision excels at anomaly detection.
QA Happens When??
Exploratory testing can be performed any time.
Rigorous, scientific approach.
Focus on customer satisfaction rather than a spec.
Equally useful before or after a release.
Just Quality
“Assurance” is a terrible word. Let’s discard it.
Quality exists, we just can’t assure or prove it.
There Is No Such
  Thing As A Formal
   Proof Of Quality.
Yet most of us would agree it exists.


I propose that “customer experience” is a better
term-of-art-than “quality.”


Though there’s no formal proof for that either.
Exploratory Testing
Addresses areas that Developer Testing can’t.
Developer Testing validates assumptions.
The Tester’s job is to invalidate assumptions.
Technology Informs
Customer Experience
Exploratory Testing requires an understanding of
the ways in which a whole system is intended to
serve a community of users.


The problem space has as much to do with
technology as it does with product requirements.
Most bugs, most of the
time, are easily nailed given
even an incomplete but
suggestive characterization
of their error conditions at
source-code level.

          —Eric S. Raymond
Source, diffs, logs.
If your QA Analysts don’t look at these, teach them.
Customer Support
Your customer support operators spend more time
talking to your users than anyone else.


Customer Support interface with users as
individuals rather than as aggregate data.
Keep the feedback loop short.
Manage Your Culture.
Effeciency To
         Thoroughness
           Trade-Off
Rapid release cycles have different risks than
slower release cycles.


But nothing about risk itself has changed.
Test Everywhere
Foreseeable errors can be worked out in dev.
Unforeseeable errors must be worked out in prod.
Fail Forward
Let go of the idea of “last stable release.”
Software exists in context.
Networks, services and people are always in flux.
Forget About
        Satisfying The
        Requirements
Watch your graphs.
Listen to your customers.
Adhere to your protocols.
Improve your product.
Questions?
@noahsussman
ns@noahsussman.com
infiniteundo.com
Further Reading
“How Google Tests Software,” James Whittaker (especially chapter 5)

“Look At Your Data,” John Rausser

“Optimizing For Developer Happiness,” Chad Dickerson

“Outages, Postmortems and Human Error,” John Allspaw

http://en.wikipedia.org/wiki/Swiss_cheese_model

“What Is Exploratory Testing?,” James Bach

“How Many Eyeballs Tame Complexity,” ESR

“The Timeless Way of Building,” Christopher Alexander

More Related Content

Viewers also liked

Continous improvement
Continous improvementContinous improvement
Continous improvementMichał Gruca
 
Continous improvement tools
Continous improvement toolsContinous improvement tools
Continous improvement toolsDEPROCESS
 
Design Patterns for Scalable Test Automation With Selenium & WebdriverIO
Design Patterns for Scalable Test Automation With Selenium & WebdriverIODesign Patterns for Scalable Test Automation With Selenium & WebdriverIO
Design Patterns for Scalable Test Automation With Selenium & WebdriverIOSauce Labs
 
Continous Improvement: How To Make it Happen
Continous Improvement: How To Make it HappenContinous Improvement: How To Make it Happen
Continous Improvement: How To Make it HappenJason Little
 
Selenium and Appium Training from Sauce Labs
Selenium and Appium Training from Sauce LabsSelenium and Appium Training from Sauce Labs
Selenium and Appium Training from Sauce LabsSauce Labs
 
Fringe Accessibility: London Web Standards
Fringe Accessibility: London Web StandardsFringe Accessibility: London Web Standards
Fringe Accessibility: London Web StandardsAdrian Roselli
 
Automation Best Practices
Automation Best PracticesAutomation Best Practices
Automation Best PracticesSauce Labs
 
Practical Tips & Tricks for Selenium Test Automation
Practical Tips & Tricks for Selenium Test AutomationPractical Tips & Tricks for Selenium Test Automation
Practical Tips & Tricks for Selenium Test AutomationSauce Labs
 
Selfish Accessibility: Government Digital Service
Selfish Accessibility: Government Digital ServiceSelfish Accessibility: Government Digital Service
Selfish Accessibility: Government Digital ServiceAdrian Roselli
 
Continuous Improvement Project
Continuous Improvement ProjectContinuous Improvement Project
Continuous Improvement ProjectDarlene Lebaste
 
5 Foolproof Business Communication Tools Every Company Needs
5 Foolproof Business Communication Tools Every Company Needs5 Foolproof Business Communication Tools Every Company Needs
5 Foolproof Business Communication Tools Every Company NeedsAxero Solutions
 
[SEN#7] Le Top 100 des entreprises qui recrutent dans le numérique
[SEN#7] Le Top 100 des entreprises qui recrutent dans le numérique[SEN#7] Le Top 100 des entreprises qui recrutent dans le numérique
[SEN#7] Le Top 100 des entreprises qui recrutent dans le numériqueFrenchWeb.fr
 
CSW2017 Peng qiu+shefang-zhong win32k -dark_composition_finnal_finnal_rm_mark
CSW2017 Peng qiu+shefang-zhong win32k -dark_composition_finnal_finnal_rm_markCSW2017 Peng qiu+shefang-zhong win32k -dark_composition_finnal_finnal_rm_mark
CSW2017 Peng qiu+shefang-zhong win32k -dark_composition_finnal_finnal_rm_markCanSecWest
 

Viewers also liked (13)

Continous improvement
Continous improvementContinous improvement
Continous improvement
 
Continous improvement tools
Continous improvement toolsContinous improvement tools
Continous improvement tools
 
Design Patterns for Scalable Test Automation With Selenium & WebdriverIO
Design Patterns for Scalable Test Automation With Selenium & WebdriverIODesign Patterns for Scalable Test Automation With Selenium & WebdriverIO
Design Patterns for Scalable Test Automation With Selenium & WebdriverIO
 
Continous Improvement: How To Make it Happen
Continous Improvement: How To Make it HappenContinous Improvement: How To Make it Happen
Continous Improvement: How To Make it Happen
 
Selenium and Appium Training from Sauce Labs
Selenium and Appium Training from Sauce LabsSelenium and Appium Training from Sauce Labs
Selenium and Appium Training from Sauce Labs
 
Fringe Accessibility: London Web Standards
Fringe Accessibility: London Web StandardsFringe Accessibility: London Web Standards
Fringe Accessibility: London Web Standards
 
Automation Best Practices
Automation Best PracticesAutomation Best Practices
Automation Best Practices
 
Practical Tips & Tricks for Selenium Test Automation
Practical Tips & Tricks for Selenium Test AutomationPractical Tips & Tricks for Selenium Test Automation
Practical Tips & Tricks for Selenium Test Automation
 
Selfish Accessibility: Government Digital Service
Selfish Accessibility: Government Digital ServiceSelfish Accessibility: Government Digital Service
Selfish Accessibility: Government Digital Service
 
Continuous Improvement Project
Continuous Improvement ProjectContinuous Improvement Project
Continuous Improvement Project
 
5 Foolproof Business Communication Tools Every Company Needs
5 Foolproof Business Communication Tools Every Company Needs5 Foolproof Business Communication Tools Every Company Needs
5 Foolproof Business Communication Tools Every Company Needs
 
[SEN#7] Le Top 100 des entreprises qui recrutent dans le numérique
[SEN#7] Le Top 100 des entreprises qui recrutent dans le numérique[SEN#7] Le Top 100 des entreprises qui recrutent dans le numérique
[SEN#7] Le Top 100 des entreprises qui recrutent dans le numérique
 
CSW2017 Peng qiu+shefang-zhong win32k -dark_composition_finnal_finnal_rm_mark
CSW2017 Peng qiu+shefang-zhong win32k -dark_composition_finnal_finnal_rm_markCSW2017 Peng qiu+shefang-zhong win32k -dark_composition_finnal_finnal_rm_mark
CSW2017 Peng qiu+shefang-zhong win32k -dark_composition_finnal_finnal_rm_mark
 

More from Noah Sussman

JavaScript Static Analysis Tools and Techniques - STP Online Session 2013
JavaScript Static Analysis Tools and Techniques - STP Online Session 2013JavaScript Static Analysis Tools and Techniques - STP Online Session 2013
JavaScript Static Analysis Tools and Techniques - STP Online Session 2013Noah Sussman
 
Continuous Improvement (GroupOn, Palo Alto 2013)
Continuous Improvement (GroupOn, Palo Alto 2013)Continuous Improvement (GroupOn, Palo Alto 2013)
Continuous Improvement (GroupOn, Palo Alto 2013)Noah Sussman
 
Jenkins data mining on the command line - Jenkins User Conference NYC 2012
Jenkins data mining on the command line - Jenkins User Conference NYC 2012Jenkins data mining on the command line - Jenkins User Conference NYC 2012
Jenkins data mining on the command line - Jenkins User Conference NYC 2012Noah Sussman
 
The user experience of CI systems - Penguicon 2012
The user experience of CI systems - Penguicon 2012The user experience of CI systems - Penguicon 2012
The user experience of CI systems - Penguicon 2012Noah Sussman
 
Selenium in the enterprise what went right and what went wrong so far - sel...
Selenium in the enterprise   what went right and what went wrong so far - sel...Selenium in the enterprise   what went right and what went wrong so far - sel...
Selenium in the enterprise what went right and what went wrong so far - sel...Noah Sussman
 

More from Noah Sussman (6)

JavaScript Static Analysis Tools and Techniques - STP Online Session 2013
JavaScript Static Analysis Tools and Techniques - STP Online Session 2013JavaScript Static Analysis Tools and Techniques - STP Online Session 2013
JavaScript Static Analysis Tools and Techniques - STP Online Session 2013
 
Continuous Improvement (GroupOn, Palo Alto 2013)
Continuous Improvement (GroupOn, Palo Alto 2013)Continuous Improvement (GroupOn, Palo Alto 2013)
Continuous Improvement (GroupOn, Palo Alto 2013)
 
Jenkins data mining on the command line - Jenkins User Conference NYC 2012
Jenkins data mining on the command line - Jenkins User Conference NYC 2012Jenkins data mining on the command line - Jenkins User Conference NYC 2012
Jenkins data mining on the command line - Jenkins User Conference NYC 2012
 
The user experience of CI systems - Penguicon 2012
The user experience of CI systems - Penguicon 2012The user experience of CI systems - Penguicon 2012
The user experience of CI systems - Penguicon 2012
 
Selenium in the enterprise what went right and what went wrong so far - sel...
Selenium in the enterprise   what went right and what went wrong so far - sel...Selenium in the enterprise   what went right and what went wrong so far - sel...
Selenium in the enterprise what went right and what went wrong so far - sel...
 
Scaling Selenium
Scaling SeleniumScaling Selenium
Scaling Selenium
 

Recently uploaded

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dashnarutouzumaki53779
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...AliaaTarek5
 
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesExploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesSanjay Willie
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Scott Andery
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 

Recently uploaded (20)

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dash
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
 
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesExploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 

Continuous improvement devops day mountain view 2012

  • 1. Continuous Improvement How rapid release cycles alter QA and testing. Noah Sussman DevopsDay Mountain View, 2012 @noahsussman #devopsdays
  • 2. The Canonical Agile Release Cycle Sprints of two weeks or more in length. Start deployment process at the end of the sprint. QA is part of the deployment process. QA must be complete before new code goes live.
  • 3. The Continuous Release Cycle Minimum viable feature set. Deployment is decoupled from release. Real-time data on how releases impact revenue. Constant tweaks to live features.
  • 4. Releasing a feature is decoupled from deploying code. David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122
  • 5. An airport without an air traffic controller. —Chad Dickerson
  • 6. This Part Really Is Different From Agile Large features are deployed piecemeal over time. Every feature is part of an A/B campaign. Dark and Admin-Only launches. Wire-Offs and Config Flags. There is no “Done Done.”
  • 7. Observed Behavior Of Complex Systems Emergent behaviors require unplanned responses. Improvements, too are discovered not designed. Users of the system have complex expectations. Such systems are never “complete.”
  • 8. QA Happens When? First of all, what is “Quality Assurance?” Authoritatively assuring that there are no defects? That’s impossible.
  • 10. Myths About Bug Detection There are a finite number of bugs. There are a finite number of detectable bugs. All severity one bugs can be found before release Software is built to specifications. At some point, software is finished.
  • 11. The Biggest Myth Bugs have complex, unpredictable causes. In fact, most errors in software are the results of incorrect assumptions made by programmers.
  • 12. Many Small Anomalies Combined The “Swiss Cheese” model of risk presents us with a strong case for prioritizing the elimination of small errors rather than focusing on the mitigation of large catastrophic failures. Unit testing is great at eliminating small errors.
  • 13. The whole time I’m programming, I’m constantly checking my assumptions. —Rasmus Lerdorf
  • 14. Resilience, Not “Quality” Readable code. Reasonable test coverage. Sane architecture. Good debugging tools. An engineering culture that values refactoring. These are measureable goals.
  • 15. Manual Testing It doesn’t always look like you think it looks. Real-Time Monitoring is the new face of testing.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 23. Watching The Graphs Etsy collects well over a quarter million metrics. Deciding which ones matter is a human problem. Everyone watches some subset of the graphs. Human vision excels at anomaly detection.
  • 24. QA Happens When?? Exploratory testing can be performed any time. Rigorous, scientific approach. Focus on customer satisfaction rather than a spec. Equally useful before or after a release.
  • 25. Just Quality “Assurance” is a terrible word. Let’s discard it. Quality exists, we just can’t assure or prove it.
  • 26. There Is No Such Thing As A Formal Proof Of Quality. Yet most of us would agree it exists. I propose that “customer experience” is a better term-of-art-than “quality.” Though there’s no formal proof for that either.
  • 27. Exploratory Testing Addresses areas that Developer Testing can’t. Developer Testing validates assumptions. The Tester’s job is to invalidate assumptions.
  • 28. Technology Informs Customer Experience Exploratory Testing requires an understanding of the ways in which a whole system is intended to serve a community of users. The problem space has as much to do with technology as it does with product requirements.
  • 29. Most bugs, most of the time, are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level. —Eric S. Raymond
  • 30. Source, diffs, logs. If your QA Analysts don’t look at these, teach them.
  • 31. Customer Support Your customer support operators spend more time talking to your users than anyone else. Customer Support interface with users as individuals rather than as aggregate data.
  • 32. Keep the feedback loop short.
  • 34. Effeciency To Thoroughness Trade-Off Rapid release cycles have different risks than slower release cycles. But nothing about risk itself has changed.
  • 35. Test Everywhere Foreseeable errors can be worked out in dev. Unforeseeable errors must be worked out in prod.
  • 36. Fail Forward Let go of the idea of “last stable release.” Software exists in context. Networks, services and people are always in flux.
  • 37. Forget About Satisfying The Requirements Watch your graphs. Listen to your customers. Adhere to your protocols. Improve your product.
  • 39. Further Reading “How Google Tests Software,” James Whittaker (especially chapter 5) “Look At Your Data,” John Rausser “Optimizing For Developer Happiness,” Chad Dickerson “Outages, Postmortems and Human Error,” John Allspaw http://en.wikipedia.org/wiki/Swiss_cheese_model “What Is Exploratory Testing?,” James Bach “How Many Eyeballs Tame Complexity,” ESR “The Timeless Way of Building,” Christopher Alexander