Making the Unstable Stable - An Intro To Testing

Cameron Presley
Cameron PresleySoftware Engineer at SentryOne
Making the Unstable Stable
An Introduction to Testing
CAMERON PRESLEY
@PCAMERONPRESLEY
HTTP://BLOG.THESOFTWAREMENTOR.COM
Making the Unstable Stable - An Intro To Testing
Outline
◦Personal Experience
◦Business Case for Testing
◦Intro to Testing
◦Next Steps
◦Additional Resources
Tale of Three Companies
Company A
Large, privately held EMR company
Hundreds of developers
Very mature codebase
Background
Team consisted of 3 developers, 2 QA
Tool that moved patient info from one hospital to another
Errors could lead to data integrity issues
Experience
Fear of refactoring code (too brittle)
Bugs kept coming back
Firefighting mode
No clear focus
What Did I Learn?
Bug count never decreased
Regressions in functionality
Firefighting all the time
Relied upon QA to find issues for us
Company B
Small publicly held company in the healthcare industry
Medical diagnostic device with custom hardware
Coming into a rewrite of the software
Background
Team consisted of 3 developers, 1 QA
Responsible for calculating test results for given samples
Errors could lead to the doctor making the wrong decision
Experience
Refactored constantly
Heavily unit tested components
Bugs never came back
Clear focus on goals
Lessons Learned
Bug count decreased and able to focus on new features
Confidence that we didn’t break anything (able to change freely)
Was it the team or was it testing?
Company C
Startup in the project management space
Help project manager plan “what-if” scenarios
Working on the initial release of the software
Background
Team consisted of 3 developers, 2 QA
Responsible for all parts of the application
Errors could lead to managers making the wrong decision
Experience
Lots of bugs
Demos would crash
◦ “Pucker factor”
Reactive, not proactive
Experience
Spent time teaching the team how and when to write tests
Product began to stabilize
Fewer bugs were being found
Experience
Shipped a massive feature with zero bugs
Confidence in the product soared
Began to focus on new features
Experience
When I left, there were over 4,000 tests
Entire codebase not covered
◦ Covered major features/functionality
Never regressed
Personal Experience
Testing allows developers to make the changes in a low-risk
fashion
Software becomes easier to change to reflect new requirements
This is pure anecdote, where’s the research?
Business Case for Testing
Making the Unstable Stable - An Intro To Testing
Background
V1 consisted of 1,000 KLOCs written over the course of 3 years
Functional testing (manual testing) was used to ensure the
product was working correctly
No automated testing was happening
◦ Ad-hoc at best
Experiment
V2 consisted of 200 KLOCs of new code and 150 KLOCs of
changed code
Mandated unit tests on new functionality
Still relied upon functional testing to ensure no regressions
Results
During code reviews, reviewers could reject changes due to
not enough tests
Tests covered 34% of functionality
With manual testing, 85% of functionality was covered
Results
Defect Severity Version 1 Version 2
Severity 1 15.5% 16.8%
Severity 2 49.8% 40.1%
Severity 3 28.7% 18.8%
Severity 4 6.0% 3.4%
21% drop in bugs at a cost of 30% more development time
Customers Increased 10x
Support Costs Increased 3x
Developers’ Perceptions
Spent less time fixing bugs
Writing tests helped with recognizing error conditions
Helped to understand inherited code
Felt more comfortable making changes to code
Testers’ Perception
Code is much higher quality
Harder to find bugs, all the obvious ones are gone
Able to spend more time looking for more complicated errors
Not finding “happy path” bugs
Making the Unstable Stable - An Intro To Testing
Code Complete
Implementation
Errors
Lack of Domain
Knowledge
Changing
Requirements
Communication
Mishaps
What are the top ways bugs are created?
Code Complete
80% of all bugs come from 20% of the code
80% of maintenance comes from 20% of the code
Code Complete
Time
Introduced
Requirements Architecture Construction System
Test
Post-Release
Requirements 1 3 5-10 10 10-100
Architecture - 1 10 15 25-100
Construction - - 1 10 10-25
Automated Testing Can Provide…
Examples of what the code should do
Built-in documentation
A way to make sure requirements are met
Common language for everyone
The Art of Unit Testing
The Art of Unit Testing
Stage Without Tests With Tests
Implementation 7 14
Integration 7 2
Testing and Bug Fixing
Testing
Fixing
Testing
Fixing
Testing
3
3
3
2
1
3
1
1
1
1
Total 26 23
Bugs Found In Production 71 11
Introduction to Testing
Introducing the Testing Triangle
Introduced by Michael Cohn in
Succeeding with Agile
Breakdown of a test suite
◦ 10% UI
◦ 20% Integration
◦ 70% Unit
Example Test
Parts of a Test (AAA)
Arrange – Get code ready to test
◦ Create dependencies, set properties, call methods, etc…
Act – Call the code that is being tested
◦ Method, non-trivial property, constructor, etc…
Assert – Did we get what we expected?
◦ Throw an exception?, Return an empty list?, Return 42?
Using AAA
Types of Automated Tests
Unit Tests
◦ Does this one piece work as expected?
Integration Tests
◦ Does this one piece work as I’d expect with others?
UI/Functional/End-to-End
◦ When performing this workflow, does the application do what I’d
expect?
Unit Tests
“A unit test is an automated piece of code that invokes a unit
of work being tested, and then checks some assumptions
about a single end result of that unit.”
- Roy Osherove (The Art of Unit Testing 2nd Edition)
Unit Tests
DOES NOT DEPEND UPON EXTERNAL FACTORS!
Typically tests a method or non-trivial property
Super fast to run (100+ per second)
Aren’t likely to change with infrastructure changes
Making the Unstable Stable - An Intro To Testing
Integration Tests
“Integration testing is testing a unit of work without having
full control over all of it and using one or more real
dependencies, such as time, network, database, threads,
number generators, and so on.”
- Roy Osherove (The Art of Unit Testing 2nd Edition)
Integration Tests
Tests how your code interacts with databases, networks, 3rd party
resources
Moderately fast to run (10+ per second)
Decently robust, but can be hard to verify correct behavior
Making the Unstable Stable - An Intro To Testing
UI Tests
Tests the application by driving the UI
Primarily used for end-to-end testing
Very slow (takes seconds per test)
Very fragile since UI changes can cause the test to break
UI Tests
Record and play-back
◦Records the interaction between user and application
◦Can be done with QA
Coded
◦Specify UI controls by ID or label to interact with
◦Primarily developer driven
Tools of the Trade
Testing Framework
Absolute minimum for writing tests
Provides a way to mark classes/methods as tests
Allows us to check if result is what we expected
◦ AreEqual, IsTrue, IsFalse, etc…
Testing Framework
Mocking Framework
Unit tests are faster than integration tests
Lot of code involves different classes collaborating together
How do we unit test these classes?
Create test dependencies that we can control
◦ Using interfaces or inheritance
Mocking Framework
Creates a lot of test code
Mocking framework uses reflection to create dependencies
Can be used to check that methods/properties were called or to stub a
response
Examples
◦ NSbustitute, Moq, Mockito
Mocking Framework
Asserting Methods Were Called
Mocking Framework
Stubbing Responses
UI Framework
Provides a mechanism to interact with UI controls
Great to provide end-to-end workflow testing
Examples
◦Test Stack White (Desktop), Selenium (Web), HP Quick Test Pro
Tools of the Trade
Testing Framework
◦ Allows us to write tests
Mocking Framework
◦ Allows us to switch out real dependencies for fake ones
UI Framework
◦ Allows us to interact with the UI components
Next Steps
Learning How to Write Tests
Sample Exercises
◦Fizz Buzz
◦String Calculator
◦Bowling Game
◦Mars Rover
Next Steps
Identify Problem Areas
Find places in your codebase that have few dependencies
◦ Easier to test and will help build confidence
Find places in your codebase that generates bugs
◦ Remember 80% of issues come from 20% of the code
◦ Might need to refactor code to make it testable
Don’t know the areas? Ask QA
Next Steps
Low Impact Testing
Don’t have to immediately write tests around everything
Start by adding tests on code that you change going forward
Just focus on bug fixes
◦ Write a test that fails because the bug exists
◦ Test should pass once the bug is fixed
Next Steps
Convincing Your Boss
Goal is to move applications to production
Remember, bugs are more expensive the later they’re found
◦ 25x more expensive if not caught before production
Every time QA finds a bug, there’s going to be churn
Puts more pressure on QA and the developers
If You’re a Manager…
Identify those on your team that know how to write tests
◦ Have those developers distribute their knowledge to others
Provide training on how to write tests
◦ Pluralsight has some great resources (free 12 month access for all MSDN holders)
◦ The Art of Unit Testing (2nd Edition) by Roy Osherove
Incorporate slack time to give developers time to learn
Remember!
Goal is to decrease time to production, not 100% code coverage
Not all applications need tests
Not every single line of code needs coverage
◦ 80% of issues come from 20% of the code
Be pragmatic, not dogmatic with testing
Want to Learn More?
10,000 foot view on testing
◦ Starting to Unit Test: Not as Hard as You Think (e-book) by Erik Dietrich
In-Depth Look at Testing
◦ The Art of Unit Testing (2nd Edition) by Roy Osherove
Testing Legacy Code
◦ Working Effectively With Legacy Code by Michael Feathers
◦ Refactoring by Martin Fowler
Additional Resources
Books
◦ Code Complete (2nd Edition) by Steve McConnell
Articles
◦ On the Effectiveness of Unit Testing at Microsoft
Presentations
◦ Unit Testing Makes Me Faster by Jeremy Clark
Videos
◦ Introduction to .NET Testing with NUnit by Jason Roberts
Thanks!
Email: Cameron@TheSoftwareMentor.com
Twitter: @PCameronPresley
Blog: http://blog.TheSoftwareMentor.com
1 of 68

Recommended

The Engineer's Playbook: Starting a New Role by
The Engineer's Playbook: Starting a New RoleThe Engineer's Playbook: Starting a New Role
The Engineer's Playbook: Starting a New RoleCameron Presley
73 views134 slides
Testing: the more you do it, the more you'll like it by
Testing: the more you do it, the more you'll like itTesting: the more you do it, the more you'll like it
Testing: the more you do it, the more you'll like itJeffrey McGuire
253 views53 slides
Pair programming demystified by
Pair programming demystifiedPair programming demystified
Pair programming demystifiedMarek Kirejczyk
1.1K views38 slides
Best pratice by
Best praticeBest pratice
Best praticeEugenio Romano
341 views22 slides
Pair Programming Presentation by
Pair Programming PresentationPair Programming Presentation
Pair Programming PresentationThoughtWorks
8.1K views30 slides
Test-Driven Development by
Test-Driven DevelopmentTest-Driven Development
Test-Driven Developmentadrianmitev
518 views22 slides

More Related Content

What's hot

Tdd by
TddTdd
Tddnitinkansal2003
418 views16 slides
Code review by
Code reviewCode review
Code reviewAleksey Solntsev
6.4K views48 slides
Common mistakes in software testing and how to overcome? by
Common mistakes in software testing and how to overcome?Common mistakes in software testing and how to overcome?
Common mistakes in software testing and how to overcome?MD ISLAM
274 views8 slides
XP Injection by
XP InjectionXP Injection
XP InjectionAleksey Solntsev
5.8K views29 slides
How to be proud when you are done by
How to be proud when you are doneHow to be proud when you are done
How to be proud when you are doneAleksey Solntsev
6K views44 slides
Extreme & pair programming Slides ppt by
Extreme & pair programming Slides pptExtreme & pair programming Slides ppt
Extreme & pair programming Slides pptMr SMAK
5.5K views23 slides

What's hot(20)

Common mistakes in software testing and how to overcome? by MD ISLAM
Common mistakes in software testing and how to overcome?Common mistakes in software testing and how to overcome?
Common mistakes in software testing and how to overcome?
MD ISLAM274 views
Extreme & pair programming Slides ppt by Mr SMAK
Extreme & pair programming Slides pptExtreme & pair programming Slides ppt
Extreme & pair programming Slides ppt
Mr SMAK5.5K views
Re-thinking Test Automation and Test Process Modelling (in pictures) by Alan Richardson
Re-thinking Test Automation and Test Process Modelling (in pictures)Re-thinking Test Automation and Test Process Modelling (in pictures)
Re-thinking Test Automation and Test Process Modelling (in pictures)
Alan Richardson5.4K views
assertYourself - Breaking the Theories and Assumptions of Unit Testing in Flex by michael.labriola
assertYourself - Breaking the Theories and Assumptions of Unit Testing in FlexassertYourself - Breaking the Theories and Assumptions of Unit Testing in Flex
assertYourself - Breaking the Theories and Assumptions of Unit Testing in Flex
michael.labriola839 views
Test driven development by Nascenia IT
Test driven developmentTest driven development
Test driven development
Nascenia IT637 views
Introduction to Test Driven Development by Michael Denomy
Introduction to Test Driven DevelopmentIntroduction to Test Driven Development
Introduction to Test Driven Development
Michael Denomy1.9K views
Teaching Kids Programming by Lynn Langit
Teaching Kids ProgrammingTeaching Kids Programming
Teaching Kids Programming
Lynn Langit586 views
Test Driven Development by ZendCon
Test Driven DevelopmentTest Driven Development
Test Driven Development
ZendCon2.1K views
@LinkingNote annotation in YATSPEC by Wojciech Bulaty
@LinkingNote annotation in YATSPEC@LinkingNote annotation in YATSPEC
@LinkingNote annotation in YATSPEC
Wojciech Bulaty4.5K views
Test Driven Development (TDD) & Continuous Integration (CI) by Fatkul Amri
Test Driven Development (TDD) & Continuous Integration (CI)Test Driven Development (TDD) & Continuous Integration (CI)
Test Driven Development (TDD) & Continuous Integration (CI)
Fatkul Amri1.7K views
Four Stages of Automated Testing by Bradley Temple by QA or the Highway
Four Stages of Automated Testing by Bradley TempleFour Stages of Automated Testing by Bradley Temple
Four Stages of Automated Testing by Bradley Temple
QA or the Highway275 views
Dev Ops Essentials Course by Use DevOps
Dev Ops Essentials CourseDev Ops Essentials Course
Dev Ops Essentials Course
Use DevOps111 views

Similar to Making the Unstable Stable - An Intro To Testing

Software presentation by
Software presentationSoftware presentation
Software presentationJennaPrengle
148 views50 slides
Future of QA by
Future of QAFuture of QA
Future of QAamitagarwal2006
480 views47 slides
Futureofqa by
FutureofqaFutureofqa
Futureofqaamitagarwal2006
260 views47 slides
Automation Concepts by
Automation ConceptsAutomation Concepts
Automation ConceptsNishant Worah
10.9K views29 slides
TDD Workshop UTN 2012 by
TDD Workshop UTN 2012TDD Workshop UTN 2012
TDD Workshop UTN 2012Facundo Farias
621 views16 slides
Myths and reality about software testing by
Myths and reality about software testingMyths and reality about software testing
Myths and reality about software testingAlisha Henderson
175 views7 slides

Similar to Making the Unstable Stable - An Intro To Testing(20)

Software presentation by JennaPrengle
Software presentationSoftware presentation
Software presentation
JennaPrengle148 views
Automation Concepts by Nishant Worah
Automation ConceptsAutomation Concepts
Automation Concepts
Nishant Worah10.9K views
Myths and reality about software testing by Alisha Henderson
Myths and reality about software testingMyths and reality about software testing
Myths and reality about software testing
Alisha Henderson175 views
Solving the 3 Biggest Questions in Continuous Testing by Perfecto by Perforce
Solving the 3 Biggest Questions in Continuous TestingSolving the 3 Biggest Questions in Continuous Testing
Solving the 3 Biggest Questions in Continuous Testing
Software Testing by AdroitLogic
Software TestingSoftware Testing
Software Testing
AdroitLogic405 views
Software Testing Interview Questions For Experienced by zynofustechnology
Software Testing Interview Questions For ExperiencedSoftware Testing Interview Questions For Experienced
Software Testing Interview Questions For Experienced
Why unit testingl by Priya Sharma
Why unit testinglWhy unit testingl
Why unit testingl
Priya Sharma1.8K views
Creating a successful continuous testing environment by Eran Kinsbruner by QA or the Highway
Creating a successful continuous testing environment by Eran KinsbrunerCreating a successful continuous testing environment by Eran Kinsbruner
Creating a successful continuous testing environment by Eran Kinsbruner
QA or the Highway184 views

More from Cameron Presley

Taking a Gamble with Functional Domain Modeling by
Taking a Gamble with Functional Domain ModelingTaking a Gamble with Functional Domain Modeling
Taking a Gamble with Functional Domain ModelingCameron Presley
152 views88 slides
Level Up Your Functional Programming Skills with LINQ by
Level Up Your Functional Programming Skills with LINQLevel Up Your Functional Programming Skills with LINQ
Level Up Your Functional Programming Skills with LINQCameron Presley
298 views84 slides
Functional Programming Through Construction : First Principles by
Functional Programming Through Construction : First PrinciplesFunctional Programming Through Construction : First Principles
Functional Programming Through Construction : First PrinciplesCameron Presley
117 views86 slides
Establishing a SOLID Foundation by
Establishing a SOLID FoundationEstablishing a SOLID Foundation
Establishing a SOLID FoundationCameron Presley
357 views91 slides
How Functional Programming Made Me a Better Developer by
How Functional Programming Made Me a Better DeveloperHow Functional Programming Made Me a Better Developer
How Functional Programming Made Me a Better DeveloperCameron Presley
454 views76 slides
How to Have Code Reviews That Developers Actually Want by
How to Have Code Reviews That Developers Actually WantHow to Have Code Reviews That Developers Actually Want
How to Have Code Reviews That Developers Actually WantCameron Presley
500 views59 slides

More from Cameron Presley(9)

Taking a Gamble with Functional Domain Modeling by Cameron Presley
Taking a Gamble with Functional Domain ModelingTaking a Gamble with Functional Domain Modeling
Taking a Gamble with Functional Domain Modeling
Cameron Presley152 views
Level Up Your Functional Programming Skills with LINQ by Cameron Presley
Level Up Your Functional Programming Skills with LINQLevel Up Your Functional Programming Skills with LINQ
Level Up Your Functional Programming Skills with LINQ
Cameron Presley298 views
Functional Programming Through Construction : First Principles by Cameron Presley
Functional Programming Through Construction : First PrinciplesFunctional Programming Through Construction : First Principles
Functional Programming Through Construction : First Principles
Cameron Presley117 views
How Functional Programming Made Me a Better Developer by Cameron Presley
How Functional Programming Made Me a Better DeveloperHow Functional Programming Made Me a Better Developer
How Functional Programming Made Me a Better Developer
Cameron Presley454 views
How to Have Code Reviews That Developers Actually Want by Cameron Presley
How to Have Code Reviews That Developers Actually WantHow to Have Code Reviews That Developers Actually Want
How to Have Code Reviews That Developers Actually Want
Cameron Presley500 views
Indy Code - Taking a Gamble With F#: Implementing Blackjack by Cameron Presley
Indy Code - Taking a Gamble With F#: Implementing BlackjackIndy Code - Taking a Gamble With F#: Implementing Blackjack
Indy Code - Taking a Gamble With F#: Implementing Blackjack
Cameron Presley727 views
How Functional Programming Made Me A Better Developer by Cameron Presley
How Functional Programming Made Me A Better DeveloperHow Functional Programming Made Me A Better Developer
How Functional Programming Made Me A Better Developer
Cameron Presley328 views
Establishing a SOLID Foundation - An Intro to Software Design by Cameron Presley
Establishing a SOLID Foundation - An Intro to Software DesignEstablishing a SOLID Foundation - An Intro to Software Design
Establishing a SOLID Foundation - An Intro to Software Design
Cameron Presley201 views

Recently uploaded

Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated... by
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...TomHalpin9
6 views29 slides
Agile 101 by
Agile 101Agile 101
Agile 101John Valentino
9 views20 slides
Fleet Management Software in India by
Fleet Management Software in India Fleet Management Software in India
Fleet Management Software in India Fleetable
11 views1 slide
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium... by
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...Lisi Hocke
35 views124 slides
Navigating container technology for enhanced security by Niklas Saari by
Navigating container technology for enhanced security by Niklas SaariNavigating container technology for enhanced security by Niklas Saari
Navigating container technology for enhanced security by Niklas SaariMetosin Oy
14 views34 slides
SAP FOR TYRE INDUSTRY.pdf by
SAP FOR TYRE INDUSTRY.pdfSAP FOR TYRE INDUSTRY.pdf
SAP FOR TYRE INDUSTRY.pdfVirendra Rai, PMP
24 views3 slides

Recently uploaded(20)

Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated... by TomHalpin9
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
TomHalpin96 views
Fleet Management Software in India by Fleetable
Fleet Management Software in India Fleet Management Software in India
Fleet Management Software in India
Fleetable11 views
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium... by Lisi Hocke
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Team Transformation Tactics for Holistic Testing and Quality (Japan Symposium...
Lisi Hocke35 views
Navigating container technology for enhanced security by Niklas Saari by Metosin Oy
Navigating container technology for enhanced security by Niklas SaariNavigating container technology for enhanced security by Niklas Saari
Navigating container technology for enhanced security by Niklas Saari
Metosin Oy14 views
DSD-INT 2023 Process-based modelling of salt marsh development coupling Delft... by Deltares
DSD-INT 2023 Process-based modelling of salt marsh development coupling Delft...DSD-INT 2023 Process-based modelling of salt marsh development coupling Delft...
DSD-INT 2023 Process-based modelling of salt marsh development coupling Delft...
Deltares7 views
FOSSLight Community Day 2023-11-30 by Shane Coughlan
FOSSLight Community Day 2023-11-30FOSSLight Community Day 2023-11-30
FOSSLight Community Day 2023-11-30
Shane Coughlan5 views
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ... by Donato Onofri
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...
Donato Onofri860 views
Airline Booking Software by SharmiMehta
Airline Booking SoftwareAirline Booking Software
Airline Booking Software
SharmiMehta6 views
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs by Deltares
DSD-INT 2023 The Danube Hazardous Substances Model - KovacsDSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
Deltares10 views
Myths and Facts About Hospice Care: Busting Common Misconceptions by Care Coordinations
Myths and Facts About Hospice Care: Busting Common MisconceptionsMyths and Facts About Hospice Care: Busting Common Misconceptions
Myths and Facts About Hospice Care: Busting Common Misconceptions
20231129 - Platform @ localhost 2023 - Application-driven infrastructure with... by sparkfabrik
20231129 - Platform @ localhost 2023 - Application-driven infrastructure with...20231129 - Platform @ localhost 2023 - Application-driven infrastructure with...
20231129 - Platform @ localhost 2023 - Application-driven infrastructure with...
sparkfabrik7 views
AI and Ml presentation .pptx by FayazAli87
AI and Ml presentation .pptxAI and Ml presentation .pptx
AI and Ml presentation .pptx
FayazAli8712 views
DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J... by Deltares
DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J...DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J...
DSD-INT 2023 3D hydrodynamic modelling of microplastic transport in lakes - J...
Deltares12 views
Advanced API Mocking Techniques by Dimpy Adhikary
Advanced API Mocking TechniquesAdvanced API Mocking Techniques
Advanced API Mocking Techniques
Dimpy Adhikary19 views
SUGCON ANZ Presentation V2.1 Final.pptx by Jack Spektor
SUGCON ANZ Presentation V2.1 Final.pptxSUGCON ANZ Presentation V2.1 Final.pptx
SUGCON ANZ Presentation V2.1 Final.pptx
Jack Spektor23 views

Making the Unstable Stable - An Intro To Testing

  • 1. Making the Unstable Stable An Introduction to Testing CAMERON PRESLEY @PCAMERONPRESLEY HTTP://BLOG.THESOFTWAREMENTOR.COM
  • 3. Outline ◦Personal Experience ◦Business Case for Testing ◦Intro to Testing ◦Next Steps ◦Additional Resources
  • 4. Tale of Three Companies
  • 5. Company A Large, privately held EMR company Hundreds of developers Very mature codebase
  • 6. Background Team consisted of 3 developers, 2 QA Tool that moved patient info from one hospital to another Errors could lead to data integrity issues
  • 7. Experience Fear of refactoring code (too brittle) Bugs kept coming back Firefighting mode No clear focus
  • 8. What Did I Learn? Bug count never decreased Regressions in functionality Firefighting all the time Relied upon QA to find issues for us
  • 9. Company B Small publicly held company in the healthcare industry Medical diagnostic device with custom hardware Coming into a rewrite of the software
  • 10. Background Team consisted of 3 developers, 1 QA Responsible for calculating test results for given samples Errors could lead to the doctor making the wrong decision
  • 11. Experience Refactored constantly Heavily unit tested components Bugs never came back Clear focus on goals
  • 12. Lessons Learned Bug count decreased and able to focus on new features Confidence that we didn’t break anything (able to change freely) Was it the team or was it testing?
  • 13. Company C Startup in the project management space Help project manager plan “what-if” scenarios Working on the initial release of the software
  • 14. Background Team consisted of 3 developers, 2 QA Responsible for all parts of the application Errors could lead to managers making the wrong decision
  • 15. Experience Lots of bugs Demos would crash ◦ “Pucker factor” Reactive, not proactive
  • 16. Experience Spent time teaching the team how and when to write tests Product began to stabilize Fewer bugs were being found
  • 17. Experience Shipped a massive feature with zero bugs Confidence in the product soared Began to focus on new features
  • 18. Experience When I left, there were over 4,000 tests Entire codebase not covered ◦ Covered major features/functionality Never regressed
  • 19. Personal Experience Testing allows developers to make the changes in a low-risk fashion Software becomes easier to change to reflect new requirements This is pure anecdote, where’s the research?
  • 20. Business Case for Testing
  • 22. Background V1 consisted of 1,000 KLOCs written over the course of 3 years Functional testing (manual testing) was used to ensure the product was working correctly No automated testing was happening ◦ Ad-hoc at best
  • 23. Experiment V2 consisted of 200 KLOCs of new code and 150 KLOCs of changed code Mandated unit tests on new functionality Still relied upon functional testing to ensure no regressions
  • 24. Results During code reviews, reviewers could reject changes due to not enough tests Tests covered 34% of functionality With manual testing, 85% of functionality was covered
  • 25. Results Defect Severity Version 1 Version 2 Severity 1 15.5% 16.8% Severity 2 49.8% 40.1% Severity 3 28.7% 18.8% Severity 4 6.0% 3.4% 21% drop in bugs at a cost of 30% more development time
  • 26. Customers Increased 10x Support Costs Increased 3x
  • 27. Developers’ Perceptions Spent less time fixing bugs Writing tests helped with recognizing error conditions Helped to understand inherited code Felt more comfortable making changes to code
  • 28. Testers’ Perception Code is much higher quality Harder to find bugs, all the obvious ones are gone Able to spend more time looking for more complicated errors Not finding “happy path” bugs
  • 30. Code Complete Implementation Errors Lack of Domain Knowledge Changing Requirements Communication Mishaps What are the top ways bugs are created?
  • 31. Code Complete 80% of all bugs come from 20% of the code 80% of maintenance comes from 20% of the code
  • 32. Code Complete Time Introduced Requirements Architecture Construction System Test Post-Release Requirements 1 3 5-10 10 10-100 Architecture - 1 10 15 25-100 Construction - - 1 10 10-25
  • 33. Automated Testing Can Provide… Examples of what the code should do Built-in documentation A way to make sure requirements are met Common language for everyone
  • 34. The Art of Unit Testing
  • 35. The Art of Unit Testing Stage Without Tests With Tests Implementation 7 14 Integration 7 2 Testing and Bug Fixing Testing Fixing Testing Fixing Testing 3 3 3 2 1 3 1 1 1 1 Total 26 23 Bugs Found In Production 71 11
  • 37. Introducing the Testing Triangle Introduced by Michael Cohn in Succeeding with Agile Breakdown of a test suite ◦ 10% UI ◦ 20% Integration ◦ 70% Unit
  • 39. Parts of a Test (AAA) Arrange – Get code ready to test ◦ Create dependencies, set properties, call methods, etc… Act – Call the code that is being tested ◦ Method, non-trivial property, constructor, etc… Assert – Did we get what we expected? ◦ Throw an exception?, Return an empty list?, Return 42?
  • 41. Types of Automated Tests Unit Tests ◦ Does this one piece work as expected? Integration Tests ◦ Does this one piece work as I’d expect with others? UI/Functional/End-to-End ◦ When performing this workflow, does the application do what I’d expect?
  • 42. Unit Tests “A unit test is an automated piece of code that invokes a unit of work being tested, and then checks some assumptions about a single end result of that unit.” - Roy Osherove (The Art of Unit Testing 2nd Edition)
  • 43. Unit Tests DOES NOT DEPEND UPON EXTERNAL FACTORS! Typically tests a method or non-trivial property Super fast to run (100+ per second) Aren’t likely to change with infrastructure changes
  • 45. Integration Tests “Integration testing is testing a unit of work without having full control over all of it and using one or more real dependencies, such as time, network, database, threads, number generators, and so on.” - Roy Osherove (The Art of Unit Testing 2nd Edition)
  • 46. Integration Tests Tests how your code interacts with databases, networks, 3rd party resources Moderately fast to run (10+ per second) Decently robust, but can be hard to verify correct behavior
  • 48. UI Tests Tests the application by driving the UI Primarily used for end-to-end testing Very slow (takes seconds per test) Very fragile since UI changes can cause the test to break
  • 49. UI Tests Record and play-back ◦Records the interaction between user and application ◦Can be done with QA Coded ◦Specify UI controls by ID or label to interact with ◦Primarily developer driven
  • 50. Tools of the Trade
  • 51. Testing Framework Absolute minimum for writing tests Provides a way to mark classes/methods as tests Allows us to check if result is what we expected ◦ AreEqual, IsTrue, IsFalse, etc…
  • 53. Mocking Framework Unit tests are faster than integration tests Lot of code involves different classes collaborating together How do we unit test these classes? Create test dependencies that we can control ◦ Using interfaces or inheritance
  • 54. Mocking Framework Creates a lot of test code Mocking framework uses reflection to create dependencies Can be used to check that methods/properties were called or to stub a response Examples ◦ NSbustitute, Moq, Mockito
  • 57. UI Framework Provides a mechanism to interact with UI controls Great to provide end-to-end workflow testing Examples ◦Test Stack White (Desktop), Selenium (Web), HP Quick Test Pro
  • 58. Tools of the Trade Testing Framework ◦ Allows us to write tests Mocking Framework ◦ Allows us to switch out real dependencies for fake ones UI Framework ◦ Allows us to interact with the UI components
  • 60. Learning How to Write Tests Sample Exercises ◦Fizz Buzz ◦String Calculator ◦Bowling Game ◦Mars Rover
  • 61. Next Steps Identify Problem Areas Find places in your codebase that have few dependencies ◦ Easier to test and will help build confidence Find places in your codebase that generates bugs ◦ Remember 80% of issues come from 20% of the code ◦ Might need to refactor code to make it testable Don’t know the areas? Ask QA
  • 62. Next Steps Low Impact Testing Don’t have to immediately write tests around everything Start by adding tests on code that you change going forward Just focus on bug fixes ◦ Write a test that fails because the bug exists ◦ Test should pass once the bug is fixed
  • 63. Next Steps Convincing Your Boss Goal is to move applications to production Remember, bugs are more expensive the later they’re found ◦ 25x more expensive if not caught before production Every time QA finds a bug, there’s going to be churn Puts more pressure on QA and the developers
  • 64. If You’re a Manager… Identify those on your team that know how to write tests ◦ Have those developers distribute their knowledge to others Provide training on how to write tests ◦ Pluralsight has some great resources (free 12 month access for all MSDN holders) ◦ The Art of Unit Testing (2nd Edition) by Roy Osherove Incorporate slack time to give developers time to learn
  • 65. Remember! Goal is to decrease time to production, not 100% code coverage Not all applications need tests Not every single line of code needs coverage ◦ 80% of issues come from 20% of the code Be pragmatic, not dogmatic with testing
  • 66. Want to Learn More? 10,000 foot view on testing ◦ Starting to Unit Test: Not as Hard as You Think (e-book) by Erik Dietrich In-Depth Look at Testing ◦ The Art of Unit Testing (2nd Edition) by Roy Osherove Testing Legacy Code ◦ Working Effectively With Legacy Code by Michael Feathers ◦ Refactoring by Martin Fowler
  • 67. Additional Resources Books ◦ Code Complete (2nd Edition) by Steve McConnell Articles ◦ On the Effectiveness of Unit Testing at Microsoft Presentations ◦ Unit Testing Makes Me Faster by Jeremy Clark Videos ◦ Introduction to .NET Testing with NUnit by Jason Roberts

Editor's Notes

  1. Software Engineer at Pilot Flying J (headquartered in Knoxville, TN) – always looking
  2. Over 5,000 employees Responsible for everything between making an appointment and being billed
  3. Allergies not being updated, blood type, etc…
  4. Overall, our bug count never decreased, our QA team was always stressed out and so was I!
  5. #2 – What if the project was cut? Or delayed?
  6. Lead to poor financial decisions that could bankrupt the company
  7. Shipped a massive feature with no bugs - Contractor had originally failed implementing this feature
  8. - Study done by Microsoft and NC state to see if adding unit tests would improve software quality - Compared two versions of software (V1 and V2) - Focused on only adding unit tests to the development process - 32 developers and 28 testers over 2 years
  9. Fundamentals of software architecture, construction, and quality metrics Advice and tips backed up by research, academia, and commercial practice
  10. Fundamentals of writing tests Tips and tricks to work with legacy applications Provides pros and cons of writing tests
  11. 1st -> For example, are those files really being copied over? Is that email being sent? Is the database being manipulated correctly?
  12. Two types of tests