This document discusses the benefits of automated testing including improved product quality through reduced defects, better documentation of requirements and specifications, and support for continuous integration and deployment. It describes different levels of testing from unit to acceptance. Unit tests should focus on intentions, have complete coverage, and be independent. Component tests cover interfaces to enable refactoring. Together, the different test types support refactoring, flexible design, and prevent bugs in production.

1. Automated testing
Why?
Markko Paas
markko.paas@gmail.com

2. goodies
levels
crosstab
lessons

3. ✓ Product quality
Frequency and severity of defects

4. ✓ Documentation
Business requirements
Technical specification

5. The authorization page
✓ should be at the correct url (3531ms)
✓ should display client_not_specified if client_id is missing as a parameter (2818ms)
✓ should display invalid_redirect_uri if wrong redirect_uri is sent as a parameter (2796ms)
✓ should display invalid_redirect_uri if client does not have a redirect_uri configured (2789ms)
✓ should redirect to the login page if the user is not logged in (2434ms)
✓ should redirect back to client with an access token on successful login (6149ms)
✓ should go back to client, when clicking on browser back button after successful login (5674ms)
When the user is already logged in
✓ should redirect back to client with an access token (1975ms)
✓ should redirect back to client with an id_token if the requested scope includes openid (2036ms)
✓ should redirect back to client with an id_token that contains the provided nonce (2150ms)
✓ should redirect back to client with a code as a query parameter (2100ms)
✓ should redirect back to client with a code and id_token if response_type includes both (2232ms)
✓ should go back to client, when clicking on browser back button (2448ms)
✓ should redirect to the login page if the login state has expired (4210ms)

6. ✓ Continuous integration and deployment
Rapid cycles = early feedback
Smaller scope = smaller risk

7. Effort
per
feature
Time
✓ Refactoring

8. Gerard Meszaros (2003)
✓ Efficiency

9. Category Business value
Product quality Frequency and severity of defects
Continuous integration and deployment Rapid cycles
Documentation
Business requirements Efficiency of team communication
Technical specification Effort for understanding the system
Development
Enables refactoring Cost of new features and bugfixes
Modular design Cost of new features and bugfixes
Reduce interactive debugging Cost of new features and bugfixes

10. Component2 Component3
C3mock
Component1
Unit1 Unit2
component test
unit test
U2mock
unit test
manual exploratory test
end-to-end test
acceptance test
minutes / hours
~10 s / ~ 10 min
~10 ms / ~ 10 s
~30 s / ~ 2 min

11. ✓ goodies
✓ levels
crosstab
lessons

12. Production
bug
prevention
Continuous
integration
and
deployment
Documentation Development support
Business
requirements
Technical
specification
Enables
refactoring
Flexible
design
Reduce
interactive
debugging
Exploratory ■
Prod-Smoke ▪ ■
Acceptance ▪ ■ ■
End-to-end ▪ ■ ▪
Component ▪ ■ ■ ■
Unit ▫ ■ ■ ▪ ■ ■

13. Unit tests - focus on describing the intention
Coverage 100%
Atomic
Independent
Aggressive maintenance
Critical code review
Production
bug
prevention
Continuous
integration
and
deployment
Documentation Development support
Business
requirements
Technical
specification
Enables
refactoring
Flexible
design
Reduce
interactive
debugging
Unit ▫ ■ ■ ▪ ■ ■

14. Component tests - focus on enabling refactoring
Cover 100% interface
Avoid referring internals
Mock dependencies
Production
bug
prevention
Continuous
integration
and
deployment
Documentation Development support
Business
requirements
Technical
specification
Enables
refactoring
Flexible
design
Reduce
interactive
debugging
Component ▪ ■ ■ ■

15. Recap
- Code review is your only chance for testing the readability of the specification
- Don’t let unit tests pile up and rot
- Acceptance tests and contract tests for business requirements tracking
- Unit tests for technical specification
- Maintain component tests interface-coverage

16. Why not?

18. References
Meszaros G. (2007). xUnit Test Patterns: Refactoring Test Code. Addison Wesley
http://xunitpatterns.com/~gerard/xpau2003-test-automation-manifesto-paper.pdf
http://xunitpatterns.com/~gerard/xunitTutorialSlides-V1.pdf
Clemson T. (2014). Testing Strategies in a Microservice Architecture
http://martinfowler.com/articles/microservice-testing/