1. How do you implement
Continuous Delivery?
Part 4: Automated Testing

2. Implementing
automated testing

3. critique the product
business facing
support programming
technology facing
Implementing
automated
testing
Agile Testing Quadrants
http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2
http://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/

4. business facing
support programming
technology facing
Implementing
automated
testing
Agile Testing Quadrants
critique the product
functional tests
prototypes
simulation
showcases
exploratory testing
usability testing
unit tests
component tests
system tests
performance tests
load tests
security tests
http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2
http://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/

5. Implementing
automated
testing
Agile Testing Quadrants
Balance your tests

6. Implementing
automated
testing
Software Testing Pyramid
Manual
Tests
End-to-end
Business facing
Localized
Technology facing
UI tests
Service tests
(API, Integration,
Component)
Unit tests
http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid

7. Implementing
automated
testing
Software Testing Pyramid
§ Tests at the bottom of the pyramid focus on
smaller sections of code, e.g. unit tests.
§ These tests are the foundation of a good test
automation strategy, they are quick to run and
there should be many of them.
§ They run at the earlier stages of the pipeline.
Unit tests

8. Implementing
automated
testing
Software Testing Pyramid
§ Tests in the middle of the pyramid cover larger
aggregation of code - components, services, etc.
§ Service tests provide many advantages of endto-end tests while avoiding UI complexities.
§ They run only after the build has passed unit
level tests.
Service
Tests

9. Implementing
automated
testing
Software Testing Pyramid
§ Tests at the top cover the "full stack” and are the
slowest to run.
§ Don’t write a test for every acceptance criteria
(antipattern), instead use a few journeys to cover
main areas of the code.
§ They run only after the build has passed both
the unit level and service level tests.
UI
tests

10. Implementing
automated
testing
Working practices
ü
Testers and developers should collaborate to write, run and maintain
tests.
ü
Siloed testing where development hands over tests to QA not only
creates long feedback loops, but also leads to testers duplicating
automated tests with manual tests.
ü
Expensive automated testing tools tend to make the feedback loop
worse. Developers should be able to run all tests, including performance
tests, to help them reproduce and diagnose any issue reported by QA.

11. Implementing
automated
testing
Anti-Pattern: Ice-cream cone
Manual Tests
UI tests
Service tests
(API, Integration,
Component)
Unit
tests
http://watirmelon.com/2012/01/31/introducing-the-software-testing-ice-cream-cone/

12. Implementing
automated
testing
Anti-Pattern: Ice-cream cone
x
Avoid inverting your test pyramid
x
Testing like this through the UI is slow and leads to brittle tests.
x
Avoid using only a UI-oriented testing tool, as that focuses effort on
writing UI-level automated tests.
x
If a bug is found by users, manual testing or high level testing, push a
test to catch that lower down the pyramid.
x
The only tests at a given level should be to test something that can't be
caught at a lower level, i.e. when testing multiple components together,
your tests should only check component integration, not each
component. That should have be done by lower-level tests.

13. Implementing
automated
testing
Challenges
q Flaky tests
q Tests take too much work to maintain
q Too much effort to add tests for legacy codebases

14. Test data
11150 23958945 8594594359
8498343 0 940938493 -9059894
892728937 90808908 323232

15. Test Data
Types of test data
q Test-specific data: This is the data that drives the behaviour
under test. It represents the specifics of the case under test.
q Test reference data: Data that needs to be there but actually
has little bearing upon the behaviour under test.
q Application reference data: Irrelevant to the behaviour
under test, but needs to be there to allow the application to
start up.

16. References
Best practices
Avoid dependencies between tests.
Rather than using database dumps, use the
application’s API to set up the correct state.
Don’t use production data.

17. Deployment Patterns
Stay tuned for Part 5…

18. go
Continuous Delivery
Learn More
See how Go can help you in
your CD journey
Deploy a great product faster.
Agile teams deliver working software early and often.
Go automates and streamlines the build-test-release
cycle for worry-free, continuous delivery of your
product.