SlideShare a Scribd company logo
1 of 43
Download to read offline
KEEPING
YOUR TESTS
LEAN
@iammeaghanlewis
meaghan@lever.co
meaghan-lewis
WHAT DOES
LEAN
MEAN ANYWAY?
VALUABLE
RELIABLE
FAST
1000+ TESTS
12 HR BUILD
100% FAILURE RATE
HOW DID WE GET
A LEAN
TEST SUITE?
NOT QUITE...
VALUABLE
Tests provide useful
information to your team
TESTS CAN GET OUTDATED
Ideas for VALUABLE tests
Re-evaluate & Analyze
● What is this test doing?
● What value is this test providing?
● Is this testing the right thing?
Quality vs Quantity
Collaborate on testing
Treat test code like
production code
RELIABLE
Tests should always have
the same result
Antipattern: Unreliable tests
EVEN ONE FAILURE MAKES A DIFFERENCE...
Ideas for RELIABLE tests
Isolate flaky tests
Isolate
flaky test
Analyze Fix
Move
back to
build
Monitor flakiness
● For each test run, record the failures
○ Test
○ Failure
○ # of times failed
Automatically rerun failures
Use specific test selectors
Independent tests
● Tests should all be hermetic
● Execution of one test does not affect others
● Run in dedicated testing environment
FAST
TESTS HAVE A SHORT FEEDBACK
LOOP
TESTS SHOULDN’T FEEL LIKE A MARATHON
UNIT &
INTEGRATION
TESTS
DEPLOY TO
STAGING UI TESTS DEPLOY TO
PRODUCTION
POST DEPLOY
TESTS
CONTINUOUS DEPLOYMENT PIPELINE
Ideas for fasts tests
Parallelize
Avoid testing in the UI
Headless tests
● XVFB ● PhantomJS
REMEMBER...
There are tradeoffs
It’s okay to start over
Testing is a team effort
THANKS!
@iammeaghanlewis
linkedin.com/in/meaghan-lewis/
meaghan@lever.co

More Related Content

What's hot

Stc chandan patary_ua_testing _ stc
Stc chandan patary_ua_testing _ stcStc chandan patary_ua_testing _ stc
Stc chandan patary_ua_testing _ stcChandan Patary
 
Agile testing practice
Agile testing practiceAgile testing practice
Agile testing practiceMary Jiang
 
DeliveryConf - Whole Team Approach to Testing in Continuous Delivery
DeliveryConf - Whole Team Approach to Testing in Continuous DeliveryDeliveryConf - Whole Team Approach to Testing in Continuous Delivery
DeliveryConf - Whole Team Approach to Testing in Continuous Deliverylisacrispin
 
Solving Flaky Automated Tests Using Machine Learning
Solving Flaky Automated Tests Using Machine LearningSolving Flaky Automated Tests Using Machine Learning
Solving Flaky Automated Tests Using Machine LearningJames Farrier
 
Teamleader testcon 2019
Teamleader testcon 2019Teamleader testcon 2019
Teamleader testcon 2019Thomas Noë
 
Agile Testing in the Enterprise 2016
Agile Testing in the Enterprise   2016Agile Testing in the Enterprise   2016
Agile Testing in the Enterprise 2016Janet Gregory
 
SCJP Preparation Tips
SCJP Preparation TipsSCJP Preparation Tips
SCJP Preparation TipsGanesh P
 
How to be proud when you are done
How to be proud when you are doneHow to be proud when you are done
How to be proud when you are doneMikalai Alimenkou
 
Shawn Wallace - Test automation in brownfield applications
Shawn Wallace - Test automation in brownfield applicationsShawn Wallace - Test automation in brownfield applications
Shawn Wallace - Test automation in brownfield applicationsQA or the Highway
 
Developer testing webinar
Developer testing webinar Developer testing webinar
Developer testing webinar PractiTest
 

What's hot (15)

Stc chandan patary_ua_testing _ stc
Stc chandan patary_ua_testing _ stcStc chandan patary_ua_testing _ stc
Stc chandan patary_ua_testing _ stc
 
Agile testing practice
Agile testing practiceAgile testing practice
Agile testing practice
 
DeliveryConf - Whole Team Approach to Testing in Continuous Delivery
DeliveryConf - Whole Team Approach to Testing in Continuous DeliveryDeliveryConf - Whole Team Approach to Testing in Continuous Delivery
DeliveryConf - Whole Team Approach to Testing in Continuous Delivery
 
Agile test
Agile testAgile test
Agile test
 
Solving Flaky Automated Tests Using Machine Learning
Solving Flaky Automated Tests Using Machine LearningSolving Flaky Automated Tests Using Machine Learning
Solving Flaky Automated Tests Using Machine Learning
 
Teamleader testcon 2019
Teamleader testcon 2019Teamleader testcon 2019
Teamleader testcon 2019
 
Agile Testing in the Enterprise 2016
Agile Testing in the Enterprise   2016Agile Testing in the Enterprise   2016
Agile Testing in the Enterprise 2016
 
SCJP Preparation Tips
SCJP Preparation TipsSCJP Preparation Tips
SCJP Preparation Tips
 
War for Agile
War for AgileWar for Agile
War for Agile
 
How to be proud when you are done
How to be proud when you are doneHow to be proud when you are done
How to be proud when you are done
 
Shawn Wallace - Test automation in brownfield applications
Shawn Wallace - Test automation in brownfield applicationsShawn Wallace - Test automation in brownfield applications
Shawn Wallace - Test automation in brownfield applications
 
XP (IT Brunch)
XP (IT Brunch)XP (IT Brunch)
XP (IT Brunch)
 
open2study
open2studyopen2study
open2study
 
Developer testing webinar
Developer testing webinar Developer testing webinar
Developer testing webinar
 
From mini waterfall to scrum
From mini waterfall to scrumFrom mini waterfall to scrum
From mini waterfall to scrum
 

Similar to Keeping your tests lean

Keeping Your Tests Lean
Keeping Your Tests LeanKeeping Your Tests Lean
Keeping Your Tests LeanMeaghan Lewis
 
Effective Unit Testing
Effective Unit TestingEffective Unit Testing
Effective Unit TestingEyal Kenig
 
Agile Testing and BDD with Gauge
Agile Testing and BDD with GaugeAgile Testing and BDD with Gauge
Agile Testing and BDD with GaugeOnur Erdogan
 
Agile Testing and BDD with Gauge
Agile Testing and BDD with GaugeAgile Testing and BDD with Gauge
Agile Testing and BDD with GaugeTestinium
 
Test automation: do we still need test specialists?
Test automation: do we still need test specialists?Test automation: do we still need test specialists?
Test automation: do we still need test specialists?Håkan Rönngren
 
Your A/B Tests are Lying to You
Your A/B Tests are Lying to YouYour A/B Tests are Lying to You
Your A/B Tests are Lying to YouJohn Clevenger
 
Your A/B Tests are Lying to You
Your A/B Tests are Lying to YouYour A/B Tests are Lying to You
Your A/B Tests are Lying to YouJohn Clevenger
 
Mutation Testing.pdf
Mutation Testing.pdfMutation Testing.pdf
Mutation Testing.pdfKeir Bowden
 
Scaling Quality by Building it in - Agile Tour Ottawa 2017
Scaling Quality by Building it in - Agile Tour Ottawa 2017Scaling Quality by Building it in - Agile Tour Ottawa 2017
Scaling Quality by Building it in - Agile Tour Ottawa 2017Maurizio Mancini
 
Unit Testing and TDD 2017
Unit Testing and TDD 2017Unit Testing and TDD 2017
Unit Testing and TDD 2017Xavi Hidalgo
 
From Gatekeeper to Partner by Kelsey Shannahan
From Gatekeeper to Partner by Kelsey ShannahanFrom Gatekeeper to Partner by Kelsey Shannahan
From Gatekeeper to Partner by Kelsey ShannahanQA or the Highway
 
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013TEST Huddle
 
STOP! You're Testing Too Much - Shawn Wallace
STOP!  You're Testing Too Much - Shawn WallaceSTOP!  You're Testing Too Much - Shawn Wallace
STOP! You're Testing Too Much - Shawn WallaceQA or the Highway
 
Stop! you're testing too much
Stop!  you're testing too muchStop!  you're testing too much
Stop! you're testing too muchShawn Wallace
 
The discipline of business experimentation
The discipline of business experimentationThe discipline of business experimentation
The discipline of business experimentationDtech Systems Co.
 
Agile Testing Agile Ottawa April 2015
Agile Testing   Agile Ottawa April 2015Agile Testing   Agile Ottawa April 2015
Agile Testing Agile Ottawa April 2015Dag Rowe
 
АННА ТИМОФІЄВА «Manual vs Automation QA vs Dev teams. Хто переможе?» Lviv QA ...
АННА ТИМОФІЄВА «Manual vs Automation QA vs Dev teams. Хто переможе?» Lviv QA ...АННА ТИМОФІЄВА «Manual vs Automation QA vs Dev teams. Хто переможе?» Lviv QA ...
АННА ТИМОФІЄВА «Manual vs Automation QA vs Dev teams. Хто переможе?» Lviv QA ...GoQA
 

Similar to Keeping your tests lean (20)

Keeping Your Tests Lean
Keeping Your Tests LeanKeeping Your Tests Lean
Keeping Your Tests Lean
 
Effective Unit Testing
Effective Unit TestingEffective Unit Testing
Effective Unit Testing
 
The Finishing Line
The Finishing LineThe Finishing Line
The Finishing Line
 
Agile Testing and BDD with Gauge
Agile Testing and BDD with GaugeAgile Testing and BDD with Gauge
Agile Testing and BDD with Gauge
 
Agile Testing and BDD with Gauge
Agile Testing and BDD with GaugeAgile Testing and BDD with Gauge
Agile Testing and BDD with Gauge
 
Test automation: do we still need test specialists?
Test automation: do we still need test specialists?Test automation: do we still need test specialists?
Test automation: do we still need test specialists?
 
Your A/B Tests are Lying to You
Your A/B Tests are Lying to YouYour A/B Tests are Lying to You
Your A/B Tests are Lying to You
 
Your A/B Tests are Lying to You
Your A/B Tests are Lying to YouYour A/B Tests are Lying to You
Your A/B Tests are Lying to You
 
Unit testing in PHP
Unit testing in PHPUnit testing in PHP
Unit testing in PHP
 
Mutation Testing.pdf
Mutation Testing.pdfMutation Testing.pdf
Mutation Testing.pdf
 
Scaling Quality by Building it in - Agile Tour Ottawa 2017
Scaling Quality by Building it in - Agile Tour Ottawa 2017Scaling Quality by Building it in - Agile Tour Ottawa 2017
Scaling Quality by Building it in - Agile Tour Ottawa 2017
 
Unit Testing and TDD 2017
Unit Testing and TDD 2017Unit Testing and TDD 2017
Unit Testing and TDD 2017
 
Testing recipes
Testing recipesTesting recipes
Testing recipes
 
From Gatekeeper to Partner by Kelsey Shannahan
From Gatekeeper to Partner by Kelsey ShannahanFrom Gatekeeper to Partner by Kelsey Shannahan
From Gatekeeper to Partner by Kelsey Shannahan
 
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013
Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013
 
STOP! You're Testing Too Much - Shawn Wallace
STOP!  You're Testing Too Much - Shawn WallaceSTOP!  You're Testing Too Much - Shawn Wallace
STOP! You're Testing Too Much - Shawn Wallace
 
Stop! you're testing too much
Stop!  you're testing too muchStop!  you're testing too much
Stop! you're testing too much
 
The discipline of business experimentation
The discipline of business experimentationThe discipline of business experimentation
The discipline of business experimentation
 
Agile Testing Agile Ottawa April 2015
Agile Testing   Agile Ottawa April 2015Agile Testing   Agile Ottawa April 2015
Agile Testing Agile Ottawa April 2015
 
АННА ТИМОФІЄВА «Manual vs Automation QA vs Dev teams. Хто переможе?» Lviv QA ...
АННА ТИМОФІЄВА «Manual vs Automation QA vs Dev teams. Хто переможе?» Lviv QA ...АННА ТИМОФІЄВА «Manual vs Automation QA vs Dev teams. Хто переможе?» Lviv QA ...
АННА ТИМОФІЄВА «Manual vs Automation QA vs Dev teams. Хто переможе?» Lviv QA ...
 

Recently uploaded

1C_PNS.pdf Philippines National standard
1C_PNS.pdf Philippines National standard1C_PNS.pdf Philippines National standard
1C_PNS.pdf Philippines National standardraffietividad53
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odishasmiwainfosol
 
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfExploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfkalichargn70th171
 
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxAndreas Kunz
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationBradBedford3
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)jennyeacort
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceBrainSell Technologies
 
英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作qr0udbr0
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringHironori Washizaki
 
Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Rob Geurden
 
Patterns for automating API delivery. API conference
Patterns for automating API delivery. API conferencePatterns for automating API delivery. API conference
Patterns for automating API delivery. API conferencessuser9e7c64
 
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...OnePlan Solutions
 
Post Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on IdentityPost Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on Identityteam-WIBU
 
Lecture # 8 software design and architecture (SDA).ppt
Lecture # 8 software design and architecture (SDA).pptLecture # 8 software design and architecture (SDA).ppt
Lecture # 8 software design and architecture (SDA).pptesrabilgic2
 
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfComparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfDrew Moseley
 
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Cizo Technology Services
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogueitservices996
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanyChristoph Pohl
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsChristian Birchler
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecturerahul_net
 

Recently uploaded (20)

1C_PNS.pdf Philippines National standard
1C_PNS.pdf Philippines National standard1C_PNS.pdf Philippines National standard
1C_PNS.pdf Philippines National standard
 
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company OdishaBalasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
Balasore Best It Company|| Top 10 IT Company || Balasore Software company Odisha
 
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdfExploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
Exploring Selenium_Appium Frameworks for Seamless Integration with HeadSpin.pdf
 
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptxUI5ers live - Custom Controls wrapping 3rd-party libs.pptx
UI5ers live - Custom Controls wrapping 3rd-party libs.pptx
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion Application
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
 
CRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. SalesforceCRM Contender Series: HubSpot vs. Salesforce
CRM Contender Series: HubSpot vs. Salesforce
 
英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作
 
Machine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their EngineeringMachine Learning Software Engineering Patterns and Their Engineering
Machine Learning Software Engineering Patterns and Their Engineering
 
Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...Simplifying Microservices & Apps - The art of effortless development - Meetup...
Simplifying Microservices & Apps - The art of effortless development - Meetup...
 
Patterns for automating API delivery. API conference
Patterns for automating API delivery. API conferencePatterns for automating API delivery. API conference
Patterns for automating API delivery. API conference
 
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...
 
Post Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on IdentityPost Quantum Cryptography – The Impact on Identity
Post Quantum Cryptography – The Impact on Identity
 
Lecture # 8 software design and architecture (SDA).ppt
Lecture # 8 software design and architecture (SDA).pptLecture # 8 software design and architecture (SDA).ppt
Lecture # 8 software design and architecture (SDA).ppt
 
Comparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdfComparing Linux OS Image Update Models - EOSS 2024.pdf
Comparing Linux OS Image Update Models - EOSS 2024.pdf
 
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
 
Ronisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited CatalogueRonisha Informatics Private Limited Catalogue
Ronisha Informatics Private Limited Catalogue
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
 
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving CarsSensoDat: Simulation-based Sensor Dataset of Self-driving Cars
SensoDat: Simulation-based Sensor Dataset of Self-driving Cars
 
Understanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM ArchitectureUnderstanding Flamingo - DeepMind's VLM Architecture
Understanding Flamingo - DeepMind's VLM Architecture
 

Keeping your tests lean